November 12, 2012

Setting Backbone View Element Properties

When you create Views in Backbone, you often use the "tagName" and "className" attributes, like so:

var itemView = new ItemView({
  model: item,
  tagName: "li",
  className: "item"
});

A Collection of these will look like:

  • Beer
  • Wine
  • Chips

  • One of the things we do all the time in Backbone is set classes on a View's DOM element reflecting properties of its model. For example, say each item on the list has one or both of the classes "inStock" and "purchased" in addition to the class "item", so you'd want your markup to look like:

  • Beer
  • Wine
  • Chips

  • In the past I've often set these classes in the View's render method, like so:

    var ItemView = Backbone.View.extend({
      render: function(){
        var itemData = this.model.toJSON()
            itemEl = $(this.el);
    
        // render the template
        itemEl.html($.mustache($("#item-tpl").html(),itemData));
    
        // set the element classes
        if(itemData.inStock){
          itemEl.addClass("inStock");
        }
        if(itemData.purchased){
          itemEl.addClass("purchased");
        }      
      }
    });
    

    While his works fine, those render methods get big fast, and the more you can do to keep them clean, the better. One thing you can do is remove the setting of those classes to the instantiation method:

    var itemClasses = "item"  // all items get this class
                    + (item.get("inStock") ? " inStock" : "")
                    + (item.get("purchased") ? " purchased" : "");
    var itemView = new ItemView({
      model: item,
      tagName: "li",
      className: itemClasses
    });
    

    ...which leaves you with a much simpler View:

    var ItemView = Backbone.View.extend({
      render: function(){
        var itemData = this.model.toJSON()
            itemEl = $(this.el);
    
        // render the template
        itemEl.html($.mustache($("#item-tpl").html(),itemData));
    
      }
    });
    

    March 30, 2012

    Management and the Hubble

    There's an interesting article making the rounds today:

    http://www.techworld.com.au/article/420036/what_went_wrong_hubble_space_telescope_what_managers_can_learn_from_it_/

    It's about the former head of NASA's Astrophysics division, Charlie Pellerin, and his experiences surrounding the development and subsequent repair of the flawed mirror in the Hubble Space Telescope. It's not a great piece--it devolves into fluff at the end--but the lessons Pellerin's story imply are universal.

    The actual cause of the problem was actually something very simple: a worker at the contractor who was building the main mirror used a shortcut to install a calibration mirror (he cut a hole in a piece of tape and introduced a burr in the mount), leading to a 1.3mm error in the location of the mirror's mount. Because the calibration mirror was used to determine the correctness of the main mirror, the main mirror eventually included that error.

    It turns out everyone knew about the error in the main mirror as soon as they put it in its mount, but came to a rationalization that the way the mirror was mounted allowed gravity to introduce errors--exactly the thing the mount was designed to prevent.

    This type of problem crops up all the time--Pellerin brings up the Challenger launch decision and KAL's famous accident rate in the early 1990's--and the list of ingredients look like every software project I've worked on for the last 15 years:
    1. Teams of smart people, who are
    2. working under pressure,
    3. to deliver something complex
    The key insight the article brings (and the reason I'm writing this) is that the social context in which people do their work is the most important determinant (Pellerin claims about 80%) of the outcome, and this is because of a simple fact:

    Smart, motivated people tend to think that individual abilities determine outcomes, but this is almost never true.

    When smart people encounter problems in project, the first place they look for solutions is inward--"how can I fix this?". Given that the social context is so important, this is actually the last place we should look. We all make this mistake.

    But more importantly, for you and me this has a profound meaning which can be hard to accept:


    Some problems are not fixable.

    As individuals, we can and often do work within the social context to solve a problem, but we've all been faced with situations where the context is out of our control for any number of reasons. And if you can't change that context, 80% of the time you can't fix the problem.

    January 4, 2012

    Flipstatus Born!

    Today I pushed out the project I worked on over my holiday break: v1.0 of flipstatus.com. It's a user-friendly, web-based version of the split-flap display project I've been working on periodically over the last few months, showing a realtime feed of social content. It's got a configuration screen so you don't have to edit any code or run it locally, and I've provided configurations for a few popular monitor resolutions (but the best way to see it is to plug it into your HTDV at 1080i over an HDMI cable).

    The idea started when I went to work on a Twitter plugin for the split-flap project and realized that a general-purpose realtime social display would be a cool tool for more than just developers. What if a restaurant owner could hang a display on his office wall which would show realtime tweets, foursquare checkins, yelp reviews, etc? Or how about a project manager seeing Trac commits on a status board?

    At the moment, I've only hooked it up to Twitter's search API, but since it uses the same plugin architecture as the split-flap project it should be relatively straightforward to write plugins for other APIs. One issue with those oAuth based APIs is that requests are normally rate-limited by the application key, which means that if a bunch of people hit the app at the same time you're going to get cut off. Not sure how to handle that.

    The next step will probably be to write an RSS plugin, since just about everything has an RSS feed. After that I'll need to build the oAuth flow handler to hook up Yelp, Foursquare, Facebook, etc. Eventually I'd like to build plugins for Twilio (SMS) and email handling, but that will require some backend architecture changes.

    For now, hook it up to your TV, put in some twitter usernames, hashtags and/or search terms, and watch it crank away.

    The app is at flipstatus.com, and as usual, this is an open-source project. Feel free to grab the code, play with it and contribute here:

    https://github.com/baspete/flipstatus

    [edit - changed github repo]

    December 19, 2011

    Virtual Solari Board Improved

    As part of the series of posts about displaying data, I've updated the virtual Solari board project again.

    It now uses a plugin framework to handle customized look and feel. You create a javascript object with properties and methods specific to your data source, like so:

    sf.display.ImageDrum = function() {
      this.order = [' ','AFL','AAL','BAW','DAL','UAE','KLM','DLH','RYR','UAL','AWE'];
    };
    
    sf.plugins.airport =  {
      
      // input: map of parameters
      // output: url
      url: function(params){
        var base_url = "data/airport_schedule.php";
        return base_url + "?" + params.serialize();
      },
    
      formatData: function(data){
        return data.response.results[0].data;
      }
    
    };

    sf.display.ImageDrum()'s order is now a series of strings instead of integers, which should be more readable. So in the above example, the CSS looks like:

    .splitflap .image span.ctrn { background-position: 0px 0px; } /* transition */
    .splitflap .image span.csp { background-position: 0px -40px; } /* blank */
    .splitflap .image span.AFL { background-position: 0px -80px; } /* aeroflot */
    .splitflap .image span.AAL { background-position: 0px -120px; } /* american */
    .splitflap .image span.BAW { background-position: 0px -160px; } /* british airways */
    .splitflap .image span.DAL { background-position: 0px -200px; } /* delta */
    .splitflap .image span.UAE { background-position: 0px -240px; } /* emirates */
    .splitflap .image span.KLM { background-position: 0px -280px; } /* klm */
    .splitflap .image span.DLH { background-position: 0px -320px; } /* lufthansa */
    .splitflap .image span.RYR { background-position: 0px -360px; } /* ryanair */
    .splitflap .image span.UAL { background-position: 0px -400px; } /* united */
    .splitflap .image span.AWE { background-position: 0px -440px; } /* us airways */
    

    But the big change here is that the project now uses backbone.js as its MVC framework. That means that the list of rows is a Collection and each row is a Model. This allows us to use Backbone's comparator function to handle row sorts. The Collection is always maintained in the correct order, and you have access to all the cool stuff you get in that framework.

    It also means that split-flap.js used only for the display stuff. In fact, there's really only one object there--sf.display--and you really only call two methods on it: initRow() and either loadRow() or loadSequentially(). The rest of the code is outside split-flap.js.

    Creating a board now looks like this:

    // generate the empty rows markup (a backbone View)
    var board = new Board;
    
    // get the sort, etc. params
    var container = $("#display1");
    var params = container.find(".chartPrefs input");
    var dataOptions = {
      "sort": container.find("input[name=sort]").val(),
      "order": container.find("input[name=order]").val()
    };
    
    // create the chart object (a backbone Collection)
    var flights = new Flights;
    flights.dataOptions = dataOptions;
    flights.url = sf.plugins.airport.url(params);
    flights.comparator = function(flight){
      if(dataOptions === "desc"){
        return -flight.get(dataOptions.sort);
      } else {
        return flight.get(dataOptions.sort);
      }
    }
    // update the chart (and set a refresh interval)
    flights.update(container);
    setInterval(function(){
      flights.update(container);
    }, 30000); // refresh interval
    

    See that "flights.comparator"? That's just an example of how you might want to handle sorting. How you structure your Collections is up to you. For example, the collection of flights looks like this:

    // This Collection is used to hold the datset for this board. 
    var Flights = Backbone.Collection.extend({
      update: function(container){
        this.fetch({
          success: function(response){
             sf.display.loadSequentially(response.toJSON(), container);
          }
        });
      },
      parse: function(json){
        return(sf.plugins.airport.formatData(json)); // normalize this data 
      }
    });
    

    You can see the plugin I showed earlier at work here where we call sf.plugins.airport.formatData(). This allows you to normalize your data formatting across APIs, which is something I've been pushing a lot lately.

    I've also created another example board to go with the airport departures board. This one uses the Weather Underground's Weather API to display some airport weather data. This one is actually slightly more complex than the airport board because it has to make several calls to the API to build the Collection.

    Anyway, check it out. As always, the project is on github here:
    https://github.com/baspete/Split-Flap

    and the live demo is here:
    http://dev.basdesign.com/split-flap/

    [edit - changed github repo]

    December 16, 2011

    Displaying Service-Based Data, Part 3

    In my previous post I wrote about normalizing service-based data to a common format. Now I'm going to talk about getting that data onto the screen.

    Continuing with our earlier example, say we've retrieved some weather data and transformed it into the following format:

    {
      response: {
        meta: { ... some stuff ... },
        results: [
          { id:'KSFO', temp_f:45, wdir:170, wspd:4, pressure:1026, trend:'+', sky:'partly cloudy' },
          { id:'KHAF', temp_f:52, wdir:0, wspd:0, pressure:1025, trend:'+', sky:'clear' },
          { id:'KSQL', temp_f:45, wdir:0, wspd:0, pressure:1025, trend:'+', sky:'partly cloudy' },
          { id:'KOAK', temp_f:45, wdir:60, wspd:6, pressure:1026, trend:'+', sky:'partly cloudy' }
        ]
      }
    }
    

    Displaying this data consists of two steps:

    1. Getting the data into the right order and length
    2. Rendering the data

    In some cases the data is already in the order we need because the API we used to retrieve it allowed us to specify sort criteria, max results, etc. But we may also be dealing with UI elements which allow the user to re-order this data, for example:

    
    
    

    If the user changes something here, we shouldn't be going back to the API with a second call just to get the same data in a different order. A much better solution is to always pass the data through a sorting function which picks up those parameters on the way to the the rendering function.

    In the past I've done this like so:

    var render = function(data){
      ... create some markup and insert into DOM ...
    };
    var sort = function(data){
      var container = $("#chart"),
          sortBy = container.find('input[name=sortBy]').val(),
          sortOrder = container.find('input[name=sortOrder]').val(),
          maxResults = container.find('input[name=maxResults]').val(),
          truncate = container.find('input[name=truncate]').val();
      ... sort/truncate/etc ...
      return sortedData;
    }
    render(sort(data.response.results), $("#chart"));
    

    This is a lousy idea, and I did it for years. The problem is that it requires both the sort() and render() functions to know enough about the DOM to find and read the UI elements specifying sort, maxResults and so on, and where they're supposed to render the resulting markup. A better solution is to do something like this:

    var params = function(container){
      return {
        "sortBy": container.find('input[name=sortBy]').val(),
        "sortOrder": container.find('input[name=sortOrder]').val(),
        "maxResults": container.find('input[name=maxResults]').val(),
        "truncate": container.find('input[name=truncate]').val()
      }
    };
    var filter = function(data, filterCriteria){
      ... sort/truncate/maxResults ...
      return sortedData;
    };
    var render = function(data){
      ... create some markup ...
      return markup;
    };
    
    var container = $("#chart");
    var chartType = container.attr("class").eq(0);
    container.append(render[chartType](filter(data.response.results, params(container)));
    

    params(), filter() and render() are all abstracted from the DOM now, and can be reused anywhere in your application, provided you keep your naming conventions the same everywhere.

    Now let's talk about that chart() function.

    In most web applications you're going to have to be able to handle more than one method for data visualization. You may be using flot for graphs, a jQuery plugin for sparklines, and Underscore.js for table markup. Each of these is likely to use a different data format, but because we've normalized our data we're starting in the same place for each of them.

    A good way to handle this is to do something like this:

    var render = {
      chart = function(data, options){
        ...format data for flot...
        var container = $("div");
        $.plot(container, formattedData, options.chartType);
        return container;
      },
      sparkline = function(data, options){
        ...format data for sparkline...
        var container = $("div");
        container.sparkline(formattedData, options);
        return container;
      },
      table = function(data, options){
        ...format data for table...
        var markup = _.template(options.template);
        return markup(data);
      }
    }
    

    Two things to note here: we're using jQuery to create a <div>, but we're not doing any DOM manipulation--we're simply returning that object to the calling function. Second in some cases we'll need to pass an "options" parameter along when calling render(). For example, when using the underscore.js "template" method, it's nice to be able to pass in your template markup from the page itself, like so:

    var container = $("#chart");
    var chartType = container.attr("class").eq(0);
    var options = {template: $('#table_template').html()};
    container.append(render[chartType](filter(data.response.results, params(container)), options);
    

    In the next post, I'll bring this all together in a working example.

    December 14, 2011

    Displaying Service-Based Data, Part 2

    In the first part of this series I talked about the importance of normalizing service-based data from different sources. The obvious question at this point is "normalized to what?", and that's the subject of this post.

    When we talk about displaying "data", we're usually talking about the visual display of quantitative information. A table. A chart. A map. Something which makes visual comparisons between discreet items possible. When we're displaying categories (as in a table) we're often called upon to sort the data according to one or more properties. In a time-series we need to make sure that the sequence in which we display values matches their order. So, more often than not the order in which items are displayed is important, and we're talking about an array for our primary data structure.

    What about the case where there's just a single item and we're displaying its properties? It's tempting to use a map, but it's almost always a bad idea. An array with one item in it works just fine and leaves you the ability to use the same structure for multiple items.

    For example, using the Weather Underground data from the previous post, a request for weather data from a single station returns:

    {
      response: { ... some stuff ... },
      location: { ... some stuff ... },
      current_observation: {
        ... some stuff ... 
      }
    }

    And the request for a list of weather stations returns:

    {
      response: {... some stuff ...},
      location: {
        ... some stuff ...
        nearby_weather_stations: {
          airport: {
            station: [
              { ... result 1 stuff ... },
              { ... result 2 stuff ... }
            ]
          },
          pws:{
           station:[
              { ... result 1 stuff ... },
              { ... result 2 stuff ... }
           ]
          }
          ... some more stuff ... 
        }
      }
    }
    

    A normalized data structure for both of these cases might look like:

    {
      response: {
        meta: { ... some stuff ... },
        results: [
          { ...result 1 stuff... },
          { ...result 2 stuff... },
          { ...result 3 stuff... },
          { ...result 4 stuff... }
        ]
      }
    }
    

    The normalization functions which would return the above data structure for the list of weather stations might look like:

    plugins.wunderground = {
    
      stations: {
        formatData: function(data){
          var airports = data.location.nearby_weather_stations.airport.station,
              pws = response.location.nearby_weather_stations.pws.station,
              i = 0,
              formattedData = {
                respone: {
                  meta: {whatever},
                  results: []
                }
              };
          for(i=0;i<airports.length;i++){
            formattedData.response.results.push(airports[i]);
          }
          for(i=0;i<pws.length;i++){
            formattedData.response.results.push(pws[i]);
          }
        return formattedData;
        }
      },
    
      station: {
        formatData: function(data){
          var results = data.current_observation,
              formattedData = {
                respone: {
                  meta: {whatever},
                    results: []
                  }
                };
          formattedData.response.push(results);
          return formattedData;
        }
      }
    
    };

    All this might seem like overkill until you realize it allows you to do something like this:

    var dataSrc = "wunderground";
    var id = "KSFO";
    
    $.ajax({
      url: plugins[dataSrc].stations.url(id),
      dataType: plugins[dataSrc].dataType,
      success: function(response){
        var data = plugins[dataSrc].stations.formatData(response);
        var stations = new Backbone.Collection;
        for (var i=0;i<data.response.results.length;i++){
          var station = new Backbone.Model; 
          station.url = plugins[dataSrc].station.url(data.response.results[i].id); 
          stations.add(station);
        }
      }
    });

    Without worrying about the format in which data is returned from a service. If the Weather Underground changes its data format, we only need to change the formatData() method in the plugin. If we go with a different data provider, we only need to write a plugin for that provider and change dataSrc.

    So what do we do once we've got this far? In the next post I'll talk about getting data from those data models onto the screen.

    Displaying Service-Based Data, Part 1

    Developers are frequently tasked with creating systems for the visual display of quantitative information (http://www.edwardtufte.com/tufte/books_vdqi). Over the years I've done a lot of these, and I've developed some ideas about what works and what doesn't. This is the first of several posts on the subject.

    To start, I think it's important to recognize that there are three fundamentally different steps in the process of getting information displayed in a web browser:
    1.  Data retrieval
    2.  Data formatting
    3.  Data display
    Historically, most information which eventually made it to a browser was either:
    •  Sent down to the browser as as part of a web page
    •  Assembled (or proxied) by the same server which served the page itself and exposed as a web service
    •  Retrieved and displayed in a separate frame
    There are still good reasons to use the first two options, including bootstrapping, caching, removal of sensitive data, and complex business logic which may be unavailable to the browser. There is never a good reason to use the third option.

    Prior to the widespread adoption of server-side support for JSONP the same origin policy meant that most web applications relied on their own servers for data services. This is no longer the case; the web developer is increasingly free to retrieve data from web services directly. From my point of view this is a fantastic development for two reasons:

    1. It frees companies to focus on their strengths. While Facebook, the USGS and Google all have browser interfaces, the most interesting work is being done by others who use the data made available through those organization's APIs. 
    2. It much more closely aligns the skillsets of the various software developers who work on a product with the parts of the stack where they work. Java devs are free to concentrate on ORMs without worrying about display logic. Front-end devs have access to those data models and they can use them to quickly build dynamic user interfaces.

    Having spent 7 years working in a Java/Velocity framework (and JSP before that--ouch!), I'm particularly keen on the second point. I've always felt that the arguments about how much rope a templating language should give you reflect the fact that server-side templating languages are a hack in the first place. They're an unfortunate byproduct of early browsers and ECMAScript1 forcing us to move a big chunk of display logic onto the server.

    In an earlier post I talked about how the big problem in front-end development is getting developers build their applications in the right order. In an environment where all the display logic is handled in the browser, creating a framework and a workflow which supports this is critical.

    An Example

    Say you were building an application in which you needed to display the current temperature in San Francisco. You'd have the option of using any of a number of publicly available APIs.

    Here's the one from the Weather Underground's Weather API:

    http://api.wunderground.com/api/d73bc72d0f231c10/conditions/q/CA/KSFO.json

    Here's the same request from the Yahoo Weather API:

    http://query.yahooapis.com/v1/public/yql?q=select%20item%20from%20weather.forecast%20where%20location%3D%2294107%22&format=json

    Note how differently those requests are formatted.

    Now take a look at the responses. The one from the Weather Underground looks like:

    {
        "response": { ... some stuff ... },
        "location": { ... a whole bunch of stuff },
        "current_observation": {
            ... a bunch of stuff ...
    
            "temp_f": 55.8,
    
            ... a bunch more stuff ...
        }
    }
    

    ... while the one from Yahoo! looks like:

    {
       "query":{
          "count":1,
          "created":"2011-12-14T21:57:51Z",
          "lang":"en-US",
          "results":{
             "channel":{
                "item":{
                   ... a bunch of stuff ...
    
                   "condition":{
                      "code":"30",
                      "date":"Wed, 14 Dec 2011 12:55 pm PST",
                      "temp":"56",
                      "text":"Partly Cloudy"
                   },
    
                   ... a bunch more stuff ...
                }
             }
          }
       }
    }
    

    Note how differently these responses are formatted. If you were absolutely sure about which one you were going to use, you could just slam this data into a javascript object and start working on displaying the data, but that's almost never the case. Your company may decide to switch providers, or you may need to support multiple providers or combine data sources.

    A much better solution is to create a plugin with two parts: a request URL formatter and a data normalization function which maps the response to common format--one that makes sense in the context of your application. Here's a simple example:

    myApp.plugins.wunderground = {
    
        url: function(station_code){
          var base_url = "http://api.wunderground.com/api/d73bc72d0f231c10/conditions/q/";
          return base_url + station_code + ".json?" + "callback=myCallback";
        },
    
        formatData: function(response){
          var formattedData = {};
          formattedData["id"] = response.current_observation.station_id;
          formattedData["temp"] = response.current_observation.temp_f;
          return formattedData;
        }
    };
    

    Once you have that in place, the process of displaying data becomes independent of the data source. I'll talk about why that's important in the second part of this series.