One of the hallmarks of a great web experience is the ability to retrace where you’ve been, either through use of the brower’s “back” button or returning to a previously viewed page.  It’s a simple html concept, but it can be challenging to implement in a single-page application as big as ours. At Personal Capital, we use Backbone.js as the foundation for our web application and it’s given us the ability to deliver a ton of feature-rich pages very quickly.  But when it comes to the tracking and retracing view states, Backbone.js only gives us the tools, not a framework to work with.

Routes, Query Params, and Internal Variables

There are three ways in which we can feed the state settings to our Backbone views:

  • Url Paths – Out of the box, Backbone gives us a router class that maps url paths to our client-side page views.  In our single-page application, we’ve segmented our Backbone views into sections, sub-sections, and sub-pages so that they mimic the feeling of a conventional site.  Using url paths helps to inform the user about the organization of our site.  The “portfolio” and “advisor” sections are good examples of where we nest multiple levels of sub-sections and sub-views within a section.
  • Query Params – We use plug-in called “backbone-query-parameters” to abstract query strings into our application.  Many of our pages/views are often data driven by a combination of factors, e.g. selected chart in Account Details with custom date range, and so sometimes it makes sense to expose these values to the url address bar so the state can easily be recreated and bookmarked.  Query params is also a great way to feed multiple state settings to the Backbone view without having to worry about their ordinal position.
  • Internal Variables – Values that we do not want to expose to the address bar and are only good for a single session are stored only in memory.  Our Account Details pages make use of internal variables, such as date range, to track and restore the state of the Backbone view when the user navigates between accounts.

Using any combination of these three options gives us complete flexibility in setting the state of our Backbone views.  But without a framework, it is difficult to implement state management consistently across a large-scale application such as ours.

View State Model Solution

From the standpoint of a Backbone view, it shouldn’t matter how state settings are provided, e.g. url paths, query params, or internal variables.  Wouldn’t it be nice if the settings provided all at once, say in a value object?

State Model

At Personal Capital, we built a state management framework, that stores state settings in a value object, which we call the “state model”, and it persists in memory through out the duration of a session.  The model is segmented into the three kinds of state tracking mechanisms and looks like this:


function CashFlowState () {
this.baseUrl = '/cash-flow';
this.path = ''
this.upStreamPath = '';

this.optionalUrlParams = [];
this.internalStateParams = [‘startDate’, ‘endDate’, ‘userAccountIds’];
this.userAccountIds = ‘all’;
}

The three aspects of our state model are:

  • Url Path – Comprised of the following properties:
    • this.baseUrl – Used to establish the base hash fragment that corresponds to the top-level sections of our Backbone application.
    • this.path – This is set by router.processRouteParams() when the url address bar changes.
    • this.upStreamPath – Used to give context to sub-views in relation to this.path.
    • this.optionalUrlParams – An array of name that correspond to what queryString variables a section expects.   As indicated in the name, these values are optional and are only defined at runtime as a property within the state model when there’s corresponding query string.
    • this.internalStateParams – An array of property names, which correspond to what state values are being track internally in our Backbone application.

The state model’s role is to be the source of truth for any changes to a section’s state settings, which can be done through either a change to the url address bar or from within our Backbone application.

Other System Actors

There three other main actors that make this whole system work:

  • processRouteParams() in the router – This method takes changes that were made to the url address bar and updates the section’s state model.  The updated state model and then passed into corresponding the section, and if appropriate, is passed downstream into the section’s child sub-sections and sub-views.
  • State datastore – Our state models are kept and referenced in a plain js object, just a like a dictionary. We use an ID convention for the property name in the datastore to reference the state model. As an example, our state model for our Portfolio section would be referenced from the state datastore as “datasource[‘portfolio’]”
  • Backbone.view extensions – Methods added to the prototype provide the following state management functionalities:
    • Backbone.View.prototype.updateView – A public method used to provide a consistent way to send state changes to the view.
    • Backbone.View.prototype.trackViewState – Responsible for storing changes in the views and synchronizing changes between the state of the application and the url address bar.
    • Backbone.View.prototype.saveInternalState – Saves view changes that are meant only for internal storage, which is used as part of the state restoration when the section is viewed.
    • Backbone.View.prototype.pathableSubViews – An array used to register this section’s sub-sections and sub-views that can react to changes to url paths.  Each element in the array is an object with the following properties:
      • nodeName – A string representing the node in the path for the corresponding sub-section or sub-view
      • subView – The Backbone sub-section or sub-view
      • Example – this.pathableSubView[0] = {subviewName: ‘subView1’, subview: SubView}
      • So when the path string is ‘subView1/subSubView1’, we can see that ‘subView1’ has a mapping to SubView
  • Backbone.View.prototype.processPathForSubView – Clones the state model, removes the 1st element in this.path string and places it in this.upstreamPath.

With this system in place, we now have a highly scalable way to manage states amongst our Backbone views.  It’s still a relative young piece of technology for us, so we welcome your thoughts and suggestions.

It’s a well-known “secret” that you can force webkit engines into hardware accelerated rendering by applying what should be non-effectual[link] css3[link] transforms[link]. The reason you care is though HTML5 mobile apps are great, they still require effort to get that native feeling for simple tasks like scrolling[link to momentum scrolling post] and animations. As a result, if you google something like “momentum scrolling html5” you’ll see posts that give you snippets like this:

*:not(html) {
     -webkit-transform: translateZ(0);
}

and variations thereof.

The main issue with css like this is that it can quickly crash your mobile app with an out of memory error by asking webkit to hardware accelerate every DOM element in the page, which it apparently wasn’t built to do. The other issue is that it’s just not obvious what else is affected when you switch on hardware acceleration all the way down the render tree.

Here’s a simple example where adding a transform modifies how webkit sizes an element:

What You Expected

Now With Extra Whitespace

And here’s the HTML/CSS to make it happen (you can see the same behavior in Chrome Mac 21.0.1180.89):

<!doctype html>
<html>

    <head>
        <style type="text/css">
            body {
                margin: 0;
            }
            #app {
                width: 100%;
                height: 100%;
                margin-left: -20px;
                background: #323232;
                -webkit-transform: translateZ(0);
            }
            .title-bar {
                position: fixed;
                top: 0;
                left: 0;
                right: 0;
                height: 48px;
                text-align: center;
                background: #323232;
                color: #FFF;
            }
            .title-bar h2 {
                margin: 8px 0;
                font-size: 26px;
            }
        </style>
    </head>

    <body>
        <div id="app">
            <div class="title-bar">
                 <h2>App Title</h2>

            </div>
        </div>
    </body>

</html>

There are three ways to fix the issue:

  1. remove “margin-left: -20px” from #app
  2. remove “width: 100%” from #app
  3. remove “-webkit-transform: translateZ(0)” from #app

Regardless of how you think this should render, the fact is that adding the webkit transform does change the sizing of the #app element in an unexpected (at least to me) way. Though easily diagnosed and fixed, it means I’ll have to spend more time scrutinizing the rendering layout leaving less time for building the actual functionality of an app, for now.

What? margin-left: -20px?

So how did I happen across this randomness? I was playing around with using Bootstrap and the built-in responsive design for mobile site development and it’s grid system uses negative margins as part of its column alignment. Hence, this post.

Writing boilerplate to change variables, re-calculate aggregate values and update views sucks and is error-prone.

De-coupleing the management of your data with whatever other parts of your application that affects makes your application more flexible, more testable, and actually easier to understand because there’s less munging of concerns within a single function.

Most of the newer Javascript frameworks have some sort of pattern for variable binding, in Backbone it’s model change events [http://backbonejs.org/#Model] , in Ember it’s controller bindings [http://emberjs.com/documentation/#toc_setting-up-bindings] and Ember.Observable [http://emberjs.com/api/classes/Ember.Observable.html], in Angular [http://angularjs.org/] it’s done declaratively but in the HTML directly and the same is true with KnockoutJS [http://knockoutjs.com/].

However, if you feel like some of these solutions are overly complex, don’t dig the get()/set() syntax or are interested in building a purpose-driven micro-framework you might be interested to know that Javascript actually supports transparent getter/setter functions just like ruby, java, actionscript and others. Here is a simple trio of functions that exposes this to you in a cross-browser manner and even notifies you of modifications done on an array (IE9+):

function watch(target, prop, handler) {
    if (target.__lookupGetter__(prop) != null) {
        return true;
    }
    var oldval = target[prop],
        newval = oldval,
        self = this,
        getter = function () {
            return newval;
        },
        setter = function (val) {
            if (Object.prototype.toString.call(val) === '[object Array]') {
                val = _extendArray(val, handler, self);
            }
            oldval = newval;
            newval = val;
            handler.call(target, prop, oldval, val);
        };
    if (delete target[prop]) { // can't watch constants
        if (Object.defineProperty) { // ECMAScript 5
            Object.defineProperty(target, prop, {
                get: getter,
                set: setter,
                enumerable: false,
                configurable: true
            });
        } else if (Object.prototype.__defineGetter__ && Object.prototype.__defineSetter__) { // legacy
            Object.prototype.__defineGetter__.call(target, prop, getter);
            Object.prototype.__defineSetter__.call(target, prop, setter);
        }
    }
    return this;
};

function unwatch(target, prop) {
    var val = target[prop];
    delete target[prop]; // remove accessors
    target[prop] = val;
    return this;
};

// Allows operations performed on an array instance to trigger bindings
function _extendArray(arr, callback, framework) {
    if (arr.__wasExtended === true) return;

    function generateOverloadedFunction(target, methodName, self) {
        return function () {
            var oldValue = Array.prototype.concat.apply(target);
            var newValue = Array.prototype[methodName].apply(target, arguments);
            target.updated(oldValue, motive);
            return newValue;
        };
    }
    arr.updated = function (oldValue, self) {
        callback.call(this, 'items', oldValue, this, motive);
    };
    arr.concat = generateOverloadedFunction(arr, 'concat', motive);
    arr.join = generateOverloadedFunction(arr, 'join', motive);
    arr.pop = generateOverloadedFunction(arr, 'pop', motive);
    arr.push = generateOverloadedFunction(arr, 'push', motive);
    arr.reverse = generateOverloadedFunction(arr, 'reverse', motive);
    arr.shift = generateOverloadedFunction(arr, 'shift', motive);
    arr.slice = generateOverloadedFunction(arr, 'slice', motive);
    arr.sort = generateOverloadedFunction(arr, 'sort', motive);
    arr.splice = generateOverloadedFunction(arr, 'splice', motive);
    arr.unshift = generateOverloadedFunction(arr, 'unshift', motive);
    arr.__wasExtended = true;

    return arr;
}

Now you can register a handler that will be called every time any part of your code updates a bound variable. For example:

var data = {
     quantity: 0
     , products:  []
}
, watcher = function(propertyName, oldValue, newValue){ … update some other pieces of the application … };

watch(data, 'quantity', watcher);
watch(data, 'products', watcher);

If you’re stepping through your javascript console as you add these you can see that after each call to watch, the properties on data are redefined as getter/setters. So now, you can do things like:

data.quantity = 7;
data.products.push('beer')

and notice that your watcher function is called automatically with the new property values. Hooray!

Now your watcher function can manage propagating the changes to the rest of your application instead of you manually doing that in every place you might update your variables. If we take this one step further, we can get a [sweet way to de-couple concerns even further by providing multiple watch functions per variable – different blog post].

This code is adapted from @eligrey’s original gist which itself is originally modeled after Gecko’s built in watch() functionality.

Frameworks such as PhoneGap or Sencha created a great promise of rapid development of fast and impressive apps across different platforms by focusing on client code reuse. We want to share a more holistic architecture across the server and client that let you go native on each client platform and still take advantage of code reuse.

The idea is simple: if you have control over both the server and the client design, you should not settle for a solution that only optimizes one.  If you “look at the whole board” and optimize the server for the clients, you won’t need to sacrifice the user experience for the sake of client code reuse.

Primary design goals for Personal Capital include:

  • Perfect balance between an feature-rich application and an intuitive navigation scheme
  • Creating the absolute best user experience; and
  • 100% accurate data presentation

With these design goals, we’re not satisfied with the defaults of each client platform, let alone defaults of shared code across multiple client platforms! At Personal Capital we have customized every component and every interaction in the pursuit of user happiness and achieving UX nirvana: a wowed and happy user each time that she uses our service.

Our Architecture

Our approach is simple: Go Native but optimize the server’s Web Services APIs for the client by following six simple tips to gain the time and flexibility we need to make our app stand out, on each platform, in its own way. We started experimenting with these rules two years ago when we created our Second Generation APIs, and optimized them as much as possible for the clients. These six principles allow us to streamline design and QA, and free up our client developers to spend the majority of their time on what they do best: making amazing user experiences.

Tip #1 Let Your Client Developers Write the Server API Definitions

API definitions are normally dictated by the server engineers, and REST enthusiasts and typical server-driven development paradigms have made it uncommon for client developers to be more than tangentially involved in server API definition before coding starts. We flipped this model on its head and relied on our client developers to drive the API definition, and as a result the server became much more attuned to how the clients expect the data to be structured, and minimized re-work later by maximizing communication early on.

Tip #2 Let Your Server Developers Take As Much Logic Out of Client Code as Possible

Avoid replicating code across many clients that you support and move as much logic as possible to the server. Having business logic in the client code is analogous to hard coding a configuration value. The pivotal point for us was realizing it’s better to have simple, thin, and pretty clients than complex, thick ones that have duplicated and hard coded business logic. Smart clients are a burden. Now each of the discussions between client and server developers is about “how can the server make the client’s code simpler?”

Tip #3 Don’t Be Afraid of Specialized APIs

Let’s just say this: it’s OK to serve the same data from more than one API. Why? Because when clients get data in the format they need, directly from the server, they can get it on screen faster by skipping complex transformations. And when the data format changes on the server, those changes are transparent to the clients. This has the added benefit of also not needing to care how the data is being used elsewhere in the client, making your client more loosely coupled and modular.

Tip #4 Let the Server Enforce Uniformity Across Clients

When it comes time to support multiple clients, the more you push responsibilities like rounding of amounts, calculation, and string formatting up to the server, the more time you will save your development and QA teams. You won’t need to spend time re-writing the same formatting and calculation rules on each client and fixing the same bugs on each platform. If it’s correct coming from the server, it will be correct everywhere.

Tip #5 No Workflow State Machines on the Clients

Likewise, when dealing with multiple clients, the more you push the complex state machines that deal with business logics into the server, the faster you will be able to iterate. For example instead of having a logic on the client that says “if user is in state x, and this is the first time that she is attempting to do y, show message z” just tell the client to show message z. All that logic can be encapsulated on the server and the API can just tell the client what to do. The time to market gained between each client writing and testing a complex flow versus each client simply responding to server flags is huge. It’s the difference between crazy nested ifs and a simple switch statement. Keep the complex state machines tucked safely in an API. Let your clients focus on display logic, and not managing business state.

Tip #6 Fast, Rich and Flexible APIs

If you follow all these tips, the payoff is huge: shorter time to market, simpler client code, less bugs. But if you want to pull this off, you must:

  • Make calling an API fast, really fast. Round-trip time of a request has to be as short as possible. This means server-side caching, is a must.
  • Create rich APIs that can deliver a lot of data in one call; this is especially important for mobile applications where the network overhead is much greater.
  • Add enough controls in the API definition to allow the client to ask for the right amount of data based on their flow. E.g. iPhone may not show transaction details and would just need the summary, but iPad and web do want to show these data. Give the clients the control to request the right data amount.
  • Gzip your responses as much as possible. The performance lift you get on mobile and web apps from this simple change are amazing!
  • Client-side caching of the API responses is just as important and reduces reliance on network stability to a great deal.

Go Native!

With loosely coupled client modules that receive pre-formatted data, and client developers that don’t need to implement tons of complex business logic, you can focus your client developers on what they do best. You’ve successfully freed up enough cycles that you can afford the extra spit and polish that will make your app stand out from the rest.

Last month we held a Meetup that we discussed these principles and how through this architecture you can reduce your client code base by not sharing client code, but rather sharing server code. You can watch the video here.

Modular application structure is great for loosely coupling code to make it more reusable and more maintainable, always has been, always will be.

However, encapsulation has typically flown in the face of cross-cutting concerns like analytics and logging, which for any production web app is absolutely essential. There are a few JavaScript AOP type libraries out there (based on a quick Google search) but in JavaScript we don’t have a class definition lookup like in ActionScript or the forceful reflection APIs that allow us to bust open private scopes like in Ruby.

The idea here is we’d like to be able to track events that happen in our application, along with the associated detailed information, while littering our application code with metrics crap as little as possible. For example, I’d much rather see this:

Analytics.track('signUpComplete');

than:

Analytics.track('signUpComplete', { username: username, browser: 'Chrome 11', donkeys: isItADonkey() });

the point being that I don’t want my application code to care about what the metrics code needs. One solution is to create and pass an accessor function that captures the scope you want to pull additional information from and pass that accessor function to the external module that needs the additional access.

In your application code:

var accessor = function(name){ try{ return eval(name); }catch(e){ /* log error */ return null; } };
Analytics.track('signUpComplete', accessor);

and in your other module’s code:

var eventsSpec = {
     'signUpComplete': {
          username: 'path.to.username'
          , browser: 'path.to.browser'
          , donkeys: 'isItADonkey()'
     }
};

var track = function(eventName, accessor){
    var capturedProperties = {};
    for(var key in eventsSpec[eventName]){
        capturedProperties[key] = accessor(eventsSpec[eventName][key]);
    }
    // fire off call to google analytics or whatever
};

While this means that your metrics code is tightly coupled to your application code and the way it is structured there isn’t really any way around that, the data lives where it lives and you need to get it. On the other hand, your application code is not tightly coupled to your metrics code, which is the goal, sweet!

Copy/paste the lines below into your browser’s console to see a full featured example of accessing properties, property chains and functions in the various closures:

var globalScopeObject = {
     prop: 'dot accessor in outer block scope'
};

var globalFunction = function() {
     return 'global function call';
};

var wrapped = function(){
     var outerBlockScope = 'this is outer block scope';
     var outerBlockScopeObject = {
          prop: 'dot accessor in outer block scope'
     };

     var outerBlockFunction = function() {
          return 'outer block function call';
     };

     return function(name){
          var innerBlockScope = 'this is inner block scope';
          var innerBlockScopeObject = {
               prop: 'dot accessor in inner block scope'
          };
          var innerBlockFunction = function() {
               return 'inner block function call';
          };

          try{
               return eval(name);
          }catch(e){
               if(typeof console != 'undefined'){
                    console.log(e);
               }
               return null;
          };
     };
};

var test = wrapped();

console.log(test('globalScope'));
console.log(test('outerBlockScope'));
console.log(test('innerBlockScope'));
console.log(test('globalScopeObject.prop'));
console.log(test('outerBlockScopeObject.prop'));
console.log(test('innerBlockScopeObject.prop'));
console.log(test('globalFunction()'));
console.log(test('outerBlockFunction()'));
console.log(test('innerBlockFunction()'));

Try out it and let me know!