Reusing Web Modules with RequireJS

why

  • Our application contains several modules that uniquely addresses a key aspect of wealth management. eg: Investment checkup, Net worth, etc. The application brings together various such aspects to present a comprehensive one-stop wealth management tool to the user.
  • How-ever each user’s needs are unique and so are the motivations to use our application. So we wanted to build landing pages that ties a user’s need to a module thus providing more context and motivation to use the module and eventually our entire offerings.
  • We went native for our mobile presence to provide great user experience and that has served us well. How-ever they are few modules of our application that are self contained with simpler ui design and fewer interactions with rest of the application. We wanted to embed such modules in our native app and determine if we could still deliver a great seamless user experience.

what

  • Build “stand-alone” versions of our web modules.
  • Publish the module with a unique url that will be used to embed it in other apps.

how

We use Requirejs as our module loader and its optimization tool to package a module and its dependencies into single units for production use. The module definition would define all its dependencies. During development phase, all the dependencies are loaded as required. How-ever in production mode, all the dependencies are packaged into a single file along with the module code to minimize our http calls and optimize load times. For eg:

A simple module definition would be as follows:

define([
 'jquery'
 , 'underscore'
 ], function($, _){

 var foo = {};
 // module code
 return foo;
});

And we reference the above module as follows:

<script data-main="scripts/foo" src="scripts/require.js"></script>

But most real-world application will have several modules and would share considerable number of framework libraries (jquery, underscore, backbone and etc) and utility libraries. And our application is no exception. So we defined a common module that would require all the common libs and in turn require this as part of our main module. The sub modules would still continue to list of all its required dependencies, but as part of the build process, we exclude them for optimization and to prevent from any shared lib being re-initialized/reset.

When the application starts, it loads the main module first and there after required modules as user navigates the application. Following is a gist of our build file is defined:

({
 modules: [
 { 
 name: 'app_main'
 }
 , {
 name: 'foo'
 , exclude:[
 'jquery'
 , 'underscore'
 , 'backbone'
 , 'services'
 , 'hbs'
 ]
 }
 ]
})

app_main will be loaded at the start of the application and if a user navigated to a foo module, only then the application will load foo. This worked great for us in keeping our main module lean as we kept adding more features (modules) until the requirement of reusing some of the modules in other apps/clients (mobile).

Fortunately for us, this did not involve lot of work and more importantly did not result in any redundant code. All we had to do was to define another main module that would include our common module and foo module. That is it. We had to refactor our common module a little bit to include only the core libs. Other shared libs were loaded as part of the main module (app_main). This way we did not load any libs that were not needed for foo module.

({
 modules: [
 { 
 name: 'app_main'
 }
 , {
 name: 'foo'
 , exclude:[
 'jquery'
 , 'underscore'
 , 'backbone'
 , 'services'
 , 'hbs'
 ]
 }
 , {
 name: 'app_foo'
 }
 ]
})
//app_main.
require(['common'], function (common) {
 require(['main']);
});

//app_foo
require(['common'], function (common) {
 require(['foo']);
});

This was just not fortune. From the very start of this app development, we not only aimed for modular code, but also to have an ability to extend any module to work by itself. Our entire MV* stack was architected that way and we are glad the returns are truly amazing.

We have a SPA that is comprised of several modules and at any time, any module can be extended to its own SPA app with minimal effort and no codes changes to the modules themselves. Thanks to the amazing requirejs optimization tool and a little foresight.

More details on how we use requirejs and its optimization tools are presented here.

reads

http://addyosmani.github.com/backbone-fundamentals

http://requirejs.org/docs/optimization.html

http://requirejs.org/docs/optimization.html#wholemultipage

http://requirejs.org/

 

2013 HTML5 Dev Conf Key Takeaways

 
  • Embedding complex SVGs into HTML
    • great talk about SVG and how box.js has used it to make its amazing viewer. apparently its more performant than the in-house chrome pdf viewer.
    • highly recommended read.
  • Scale and Performance: Data visualizations in modern browser
    • while the earlier talk was about not using canvas, this was all about using canvas and its might.
    • very inspiring to what folks have done and seems to be most of them are in publishing business (NYT and various media companies)
    • this and couple other similar talks strongly advocate use of D3.js for visualizations
    • Learned about Datawrapper, a tool to create simple, embeddable and interactive charts.
  • Constraint validation: Native client side validation web forms
    • talks about native validation that is available and supported in most browsers
  • Transforming the presentation of official statistics
    • amazing infographics and very inspiring to see huge amounts of data represented in easy to understand, interactive graphics.
  • ReactJS
    • new mv* js library from Facebook … oh what !!
    • its pretty much against everything that the current set of mv* libraries advocates
    • no templates, re-rendering your whole application when data changes… what??
    • and that is why it is a interesting read…never listen to anybody that says “DONT RE-INVENT THE WHEEL”
  • JS Inconsistencies Across Browsers
    • very illuminating and gives you a brief insight into how browsers work and why the inconsistencies.
    • talk about event loop, iframes and xhr gives insight into how browser processes them and advocates good coding practices when using them.
  • Continuous Delivery for JS Apps
    • the highlight of the conference for me. great speaker and very motivating
    • key takeaway: AUTOMATION

State Tracking with Backbone.js

One of the hallmarks of a great web experience is the ability to retrace where you’ve been, either through use of the brower’s “back” button or returning to a previously viewed page.  It’s a simple html concept, but it can be challenging to implement in a single-page application as big as ours. At Personal Capital, we use Backbone.js as the foundation for our web application and it’s given us the ability to deliver a ton of feature-rich pages very quickly.  But when it comes to the tracking and retracing view states, Backbone.js only gives us the tools, not a framework to work with.

Routes, Query Params, and Internal Variables

There are three ways in which we can feed the state settings to our Backbone views:

  • Url Paths – Out of the box, Backbone gives us a router class that maps url paths to our client-side page views.  In our single-page application, we’ve segmented our Backbone views into sections, sub-sections, and sub-pages so that they mimic the feeling of a conventional site.  Using url paths helps to inform the user about the organization of our site.  The “portfolio” and “advisor” sections are good examples of where we nest multiple levels of sub-sections and sub-views within a section.
  • Query Params – We use plug-in called “backbone-query-parameters” to abstract query strings into our application.  Many of our pages/views are often data driven by a combination of factors, e.g. selected chart in Account Details with custom date range, and so sometimes it makes sense to expose these values to the url address bar so the state can easily be recreated and bookmarked.  Query params is also a great way to feed multiple state settings to the Backbone view without having to worry about their ordinal position.
  • Internal Variables – Values that we do not want to expose to the address bar and are only good for a single session are stored only in memory.  Our Account Details pages make use of internal variables, such as date range, to track and restore the state of the Backbone view when the user navigates between accounts.

Using any combination of these three options gives us complete flexibility in setting the state of our Backbone views.  But without a framework, it is difficult to implement state management consistently across a large-scale application such as ours.

View State Model Solution

From the standpoint of a Backbone view, it shouldn’t matter how state settings are provided, e.g. url paths, query params, or internal variables.  Wouldn’t it be nice if the settings provided all at once, say in a value object?

State Model

At Personal Capital, we built a state management framework, that stores state settings in a value object, which we call the “state model”, and it persists in memory through out the duration of a session.  The model is segmented into the three kinds of state tracking mechanisms and looks like this:


function CashFlowState () {
this.baseUrl = '/cash-flow';
this.path = ''
this.upStreamPath = '';

this.optionalUrlParams = [];
this.internalStateParams = ['startDate', 'endDate', 'userAccountIds'];
this.userAccountIds = ‘all’;
}

The three aspects of our state model are:

  • Url Path – Comprised of the following properties:
    • this.baseUrl – Used to establish the base hash fragment that corresponds to the top-level sections of our Backbone application.
    • this.path – This is set by router.processRouteParams() when the url address bar changes.
    • this.upStreamPath – Used to give context to sub-views in relation to this.path.
    • this.optionalUrlParams – An array of name that correspond to what queryString variables a section expects.   As indicated in the name, these values are optional and are only defined at runtime as a property within the state model when there’s corresponding query string.
    • this.internalStateParams – An array of property names, which correspond to what state values are being track internally in our Backbone application.

The state model’s role is to be the source of truth for any changes to a section’s state settings, which can be done through either a change to the url address bar or from within our Backbone application.

Other System Actors

There three other main actors that make this whole system work:

  • processRouteParams() in the router – This method takes changes that were made to the url address bar and updates the section’s state model.  The updated state model and then passed into corresponding the section, and if appropriate, is passed downstream into the section’s child sub-sections and sub-views.
  • State datastore – Our state models are kept and referenced in a plain js object, just a like a dictionary. We use an ID convention for the property name in the datastore to reference the state model. As an example, our state model for our Portfolio section would be referenced from the state datastore as “datasource[‘portfolio’]”
  • Backbone.view extensions – Methods added to the prototype provide the following state management functionalities:
    • Backbone.View.prototype.updateView – A public method used to provide a consistent way to send state changes to the view.
    • Backbone.View.prototype.trackViewState – Responsible for storing changes in the views and synchronizing changes between the state of the application and the url address bar.
    • Backbone.View.prototype.saveInternalState – Saves view changes that are meant only for internal storage, which is used as part of the state restoration when the section is viewed.
    • Backbone.View.prototype.pathableSubViews – An array used to register this section’s sub-sections and sub-views that can react to changes to url paths.  Each element in the array is an object with the following properties:
      • nodeName – A string representing the node in the path for the corresponding sub-section or sub-view
      • subView – The Backbone sub-section or sub-view
      • Example – this.pathableSubView[0] = {subviewName: ‘subView1′, subview: SubView}
      • So when the path string is ‘subView1/subSubView1′, we can see that ‘subView1′ has a mapping to SubView
  • Backbone.View.prototype.processPathForSubView – Clones the state model, removes the 1st element in this.path string and places it in this.upstreamPath.

With this system in place, we now have a highly scalable way to manage states amongst our Backbone views.  It’s still a relative young piece of technology for us, so we welcome your thoughts and suggestions.

Mobile Webkit CSS Sizing Bug: Hardware Acceleration Edition

It’s a well-known “secret” that you can force webkit engines into hardware accelerated rendering by applying what should be non-effectual[link] css3[link] transforms[link]. The reason you care is though HTML5 mobile apps are great, they still require effort to get that native feeling for simple tasks like scrolling[link to momentum scrolling post] and animations. As a result, if you google something like “momentum scrolling html5″ you’ll see posts that give you snippets like this:

*:not(html) {
     -webkit-transform: translateZ(0);
}

and variations thereof.

The main issue with css like this is that it can quickly crash your mobile app with an out of memory error by asking webkit to hardware accelerate every DOM element in the page, which it apparently wasn’t built to do. The other issue is that it’s just not obvious what else is affected when you switch on hardware acceleration all the way down the render tree.

Here’s a simple example where adding a transform modifies how webkit sizes an element:

What You Expected

Now With Extra Whitespace

And here’s the HTML/CSS to make it happen (you can see the same behavior in Chrome Mac 21.0.1180.89):

<!doctype html>
<html>

    <head>
        <style type="text/css">
            body {
                margin: 0;
            }
            #app {
                width: 100%;
                height: 100%;
                margin-left: -20px;
                background: #323232;
                -webkit-transform: translateZ(0);
            }
            .title-bar {
                position: fixed;
                top: 0;
                left: 0;
                right: 0;
                height: 48px;
                text-align: center;
                background: #323232;
                color: #FFF;
            }
            .title-bar h2 {
                margin: 8px 0;
                font-size: 26px;
            }
        </style>
    </head>

    <body>
        <div id="app">
            <div class="title-bar">
                 <h2>App Title</h2>

            </div>
        </div>
    </body>

</html>

There are three ways to fix the issue:

  1. remove “margin-left: -20px” from #app
  2. remove “width: 100%” from #app
  3. remove “-webkit-transform: translateZ(0)” from #app

Regardless of how you think this should render, the fact is that adding the webkit transform does change the sizing of the #app element in an unexpected (at least to me) way. Though easily diagnosed and fixed, it means I’ll have to spend more time scrutinizing the rendering layout leaving less time for building the actual functionality of an app, for now.

What? margin-left: -20px?

So how did I happen across this randomness? I was playing around with using Bootstrap and the built-in responsive design for mobile site development and it’s grid system uses negative margins as part of its column alignment. Hence, this post.

JS Hacks: Dead Simple Javascript Variable Change Watchers

Writing boilerplate to change variables, re-calculate aggregate values and update views sucks and is error-prone.

De-coupleing the management of your data with whatever other parts of your application that affects makes your application more flexible, more testable, and actually easier to understand because there’s less munging of concerns within a single function.

Most of the newer Javascript frameworks have some sort of pattern for variable binding, in Backbone it’s model change events [http://backbonejs.org/#Model] , in Ember it’s controller bindings [http://emberjs.com/documentation/#toc_setting-up-bindings] and Ember.Observable [http://emberjs.com/api/classes/Ember.Observable.html], in Angular [http://angularjs.org/] it’s done declaratively but in the HTML directly and the same is true with KnockoutJS [http://knockoutjs.com/].

However, if you feel like some of these solutions are overly complex, don’t dig the get()/set() syntax or are interested in building a purpose-driven micro-framework you might be interested to know that Javascript actually supports transparent getter/setter functions just like ruby, java, actionscript and others. Here is a simple trio of functions that exposes this to you in a cross-browser manner and even notifies you of modifications done on an array (IE9+):

function watch(target, prop, handler) {
    if (target.__lookupGetter__(prop) != null) {
        return true;
    }
    var oldval = target[prop],
        newval = oldval,
        self = this,
        getter = function () {
            return newval;
        },
        setter = function (val) {
            if (Object.prototype.toString.call(val) === '[object Array]') {
                val = _extendArray(val, handler, self);
            }
            oldval = newval;
            newval = val;
            handler.call(target, prop, oldval, val);
        };
    if (delete target[prop]) { // can't watch constants
        if (Object.defineProperty) { // ECMAScript 5
            Object.defineProperty(target, prop, {
                get: getter,
                set: setter,
                enumerable: false,
                configurable: true
            });
        } else if (Object.prototype.__defineGetter__ && Object.prototype.__defineSetter__) { // legacy
            Object.prototype.__defineGetter__.call(target, prop, getter);
            Object.prototype.__defineSetter__.call(target, prop, setter);
        }
    }
    return this;
};

function unwatch(target, prop) {
    var val = target[prop];
    delete target[prop]; // remove accessors
    target[prop] = val;
    return this;
};

// Allows operations performed on an array instance to trigger bindings
function _extendArray(arr, callback, framework) {
    if (arr.__wasExtended === true) return;

    function generateOverloadedFunction(target, methodName, self) {
        return function () {
            var oldValue = Array.prototype.concat.apply(target);
            var newValue = Array.prototype[methodName].apply(target, arguments);
            target.updated(oldValue, motive);
            return newValue;
        };
    }
    arr.updated = function (oldValue, self) {
        callback.call(this, 'items', oldValue, this, motive);
    };
    arr.concat = generateOverloadedFunction(arr, 'concat', motive);
    arr.join = generateOverloadedFunction(arr, 'join', motive);
    arr.pop = generateOverloadedFunction(arr, 'pop', motive);
    arr.push = generateOverloadedFunction(arr, 'push', motive);
    arr.reverse = generateOverloadedFunction(arr, 'reverse', motive);
    arr.shift = generateOverloadedFunction(arr, 'shift', motive);
    arr.slice = generateOverloadedFunction(arr, 'slice', motive);
    arr.sort = generateOverloadedFunction(arr, 'sort', motive);
    arr.splice = generateOverloadedFunction(arr, 'splice', motive);
    arr.unshift = generateOverloadedFunction(arr, 'unshift', motive);
    arr.__wasExtended = true;

    return arr;
}

Now you can register a handler that will be called every time any part of your code updates a bound variable. For example:

var data = {
     quantity: 0
     , products:  []
}
, watcher = function(propertyName, oldValue, newValue){ … update some other pieces of the application … };

watch(data, 'quantity', watcher);
watch(data, 'products', watcher);

If you’re stepping through your javascript console as you add these you can see that after each call to watch, the properties on data are redefined as getter/setters. So now, you can do things like:

data.quantity = 7;
data.products.push('beer')

and notice that your watcher function is called automatically with the new property values. Hooray!

Now your watcher function can manage propagating the changes to the rest of your application instead of you manually doing that in every place you might update your variables. If we take this one step further, we can get a [sweet way to de-couple concerns even further by providing multiple watch functions per variable - different blog post].

This code is adapted from @eligrey’s original gist which itself is originally modeled after Gecko’s built in watch() functionality.