The Evolution of a Startup: Web Technologies

This a detailed post describing Personal Capital’s transition from Flex to Backbone.js+RequireJS including the reasoning behind it and the experience of doing it.

By now, it’s clear to anyone who’s actively involved in the web space, including consumers, who’s come out on top in the Flash vs Rest of the World showdown. So, as Adobe transitions into marketing some HTML animation apps after ditching Flex and abandoning Flash Player for mobile browsers, it might be interesting to take a look at why on earth our web product is currently Flex based, and why it won’t be much longer.

Use it or lose it – your market share that is

Between 2009 and 2012 marked at least two dramatic shifts in popular technology we deal with day-to-day, which is not all that unusual for technology in general, but in terms of front-end web development it’s been fast and huge. These two big shifts I’m talking about are the decline of Internet Explorer on the one hand and Flash on the other.

Ironically, the same folks that are relegating both technologies are also now currently trading punches: Apple and Google.

Apple got real busy selling iOS devices and banning Flash from them while telling the world, every day, that they were doing it. Apple was also hard at work gaining an extra 10% PC market share with OS X which doesn’t run Internet Explorer at all. In terms of consumer perception, I’d say it was all Apple. But really, in terms of enabling this new generation of technology, Google’s Chrome and V8 teams deserve 99% of the credit.

For your consideration please:

That’s a pretty phenomenal eating of Microsoft’s lunch as Chrome goes from 0 to 34% and IE goes from 69 to 33% (and thankfully 17% of that to IE9). That leaves a big fat 2% for Firefox and Safari to play with. So there’s probably a million reasons why that played out the way it did, but in my mind, the single biggest factor that drove this current round of web innovation is the javascript performance war that Google’s V8 engine started.

Back in September of 2010, when we first started working on the Personal Capital web product you see today, the shiny new IE8 clocked in at 4989.7ms running the SunSpider javascript performance benchmark. This morning, Chrome on my Macbook Air clocked in at a respectable 243ms. The slope of the line on that graph is steeper than the financial cliff we stumbled off of in 2008. It’s the reason we can have smooth animations in javascript and data-tables that sort themselves in milliseconds. Chrome’s speed and fidelity is the reason we’re even talking about this at all, not to mention all the work the Chrome team has put in along with others to get HTML5 and CSS3 out the door. That’s huge. As Javascript developers in 2012 we’ve got great performance and a blossoming feature set.

In 2010 we had neither; which is the void that Adobe Flash Player filled. Although I can’t find updated numbers for Flash Player penetration today (surprise), the 2010 Flash Player 10.1 had 87.4% (more than IE) of the US browser market covered with a powerful VM that could execute and render at high fidelity at the same time. That March, Adobe also released Flex 4 which is a great framework with a lot of thought put into separation of concerns and mature patterns like state management, complex data collections and intelligent view (re)rendering. The icing, though, were the 4th gen micro-frameworks that sat along side it. Frameworks like Swiz and Robotlegs made AOP and IoC first class citizens in the Flash/Flex world and eliminated hundreds of lines of boilerplate and decoupled code in the process.

That’s why our web product is written in Flex. It allowed us to create a highly interactive browser-based product quickly, at high quality, to our design specifications and deliver it to almost everyone while building it on a solid software architecture. Until Adobe ditched it.

You know, one thing that Flex never really had going for it was a great open source developer community, and I’m not quite sure why that is. We had plenty of good examples from python to ruby to even the jQuery plugin ecosystem. So when it didn’t make sense financially for Adobe to continue to drive Flex, there wasn’t enough of a community there to catch its fall and there wasn’t enough of one there to prevent it in the first place.

So, in 2012, Flex has been dumped unceremoniously into the Apache Foundation’s lap and is being valiantly lipo-sucked by the die-hards in the Spoon Project. Flash Player has been relegated to a video streaming and gaming platform who’s currently trying to integrate OpenGL. Internet Explorer, for its part, might make a comeback as IE10 purports to finally be on par with modern browsers both in terms of performance and a standardized feature set but I doubt Google or Mozilla is interested in sitting around and letting that happen.

The takeaway here is simple and it’s something every developer knows in their heart: you’re never done until you have no more customers. Microsoft and Adobe thought they were done and they both used their dominance to excuse themselves from continually delighting theirs so Google and Apple came in and did it for them. And they did a damn good job.

But enough of the history lesson! Let’s dive into what we’re actually doing now with Javascript and HTML5 and where we’d like to go in the future.

Our Javascript App Design Refresh

We started converting parts of our web application from Flex to Javascript around September of 2011, first as a POC, then for production but just on a simple jQuery enhanced prototype based architecture. At the same time, we were doing the research necessary in order to completely rebuild our entire development stack from scratch. Not only were we looking to maintain use of architectural patterns like dependency injection, event mediation and data binding, but we also needed to figure out how we were going to test, package and ship our software.

Here’s the current setup:

Application Framework

Backbone.js provides just enough framework to be useful but not so much that we can’t cultivate our architecture in the way we see fit. We take advantage of the router, propagated model/collection updates and simplified view setup. We’ve extended collections to do smart merges from our API backend, implemented client-side data caching for data injection (think dependency injection but with data) and have a set of rules to determine how we break down designs into individual view instances. We compared all the frameworks currently listed on TodoMVC and were impressed with how the platform is evolving, but it came down to Ember and Backbone at the time. We like the Backbone community and the learning curve although we’re aware we’ll eventually experience some growing pains. We’re closely following projects like Aura, Marionette, Thorax, and Chaplin. A more detailed write-up of our comparison here warrants another post and we are currently also experimenting with AngularJS for some more advanced UI components.

Architectural Framework

RequireJS is not just a script loader, it’s really dependency injection and gives you nodejs-ish module encapsulation in the browser. Dependency injection is important for efficiency without singletons or long-winded model locator syntax, for isolating units under test and promotes greater code reuse because of the loose coupling. The RequireJS project has the best momentum out of the alternatives we evaluated in our opinion and their shim support iced the cake. Read more about it here.

View Templating & Layout

Handlebars is a clear favorite here because even though we started out using Twitter’s Hogan implementation of Mustache we found the extra bit of flexibility worthwhile to people experienced in developing complicated UIs. Handlebars’ registerHelper mechanism is also a much cleaner way to promote code reuse of view transforms than the function mechanism in Mustache proper. And given the fact that we aren’t going with a framework which enforces a view language like AngularJS or Knockout there really isn’t anything better for dead-simple view definition.

Visual Design & Styling

Compass (and SASS) was chosen because of our desire to empower our design team more in the actual implementation of products; our goal being to provide the HTML structure and JS behavior and let the more opinionated people perfect the pixels. Given the broad library of mixins that create cross-browser CSS and the fact that SASS doesn’t have any significant disadvantages compared to LESS or Stylus, Compass was a natural choice for us. Currently, developers use Scout for compilation but this will eventually be integrated into our build process.

Unit Testing

SinonJS with Chai extensions enables simple and powerful behavior expression. If you’re used to BDD testing in Ruby the patterns here will be familiar. It’s a library that takes full advantage of the language it is written as well as the experience of other communities in evolving test frameworks. You get to write easy to read unit tests for almost anything without much pain. Seriously, if you haven’t been writing unit tests up until now, go check it out.

Build Process

Hudson is our default continuous integration server because we were already using it. While there are other choices out there, it’s got a great community of plugins and it’s quite impressive how well it’s held up building, testing and deploying everything from ActiveX plugins, WAR files and Flex projects to our Javascript applications. We use the RequireJS optimizer to concatenate and uglify our applications prior to release.

Graphing

Raphaël could use some documentation love but that doesn’t take away from its excellence as a cross-browser vector graphics library. All the HTML graphs you see in our site are custom Raphaël plugins that are really close to the heart of our value proposition. We want to give you the best insight into your finances possible and oftentimes that manifests itself as a highly interactive, highly custom visualizations that we hope bring you happiness and prosperity. We also seriously evaluated Highcharts and the graphing extension to Raphaël but ultimately our passion for numbers (and a great product team that can manipulate them) drove us to pick something productive yet 100% customizable. [TODO: link to Justin’s Raphael post here]

A Quick Note About Debugging

The visual debugger in Flash Builder was the single biggest selling point for Flash Builder, but it required a debug version of Flash Player to be installed and Flash Builder (Eclipse) to be running. Yikes.

Now I want to take a moment to wholeheartedly thank the Firebug team for pioneering the in-browser debugging console back in 2006 and doing it right. These days the Chrome Developer Tools and Safari Web Inspector are top notch and even IE9+ has some semblance of a useful developer console which allows breakpointing, runtime manipulation and analysis AND is deployed on customer machines all without having to boot up a separate program. We’re even able to enable rendering hints in webkit browsers. This is one area in which I feel Javascript development is clearly superior to any other technology I’ve worked with.

Functional Testing

Unfortunately, we haven’t nailed this down yet and both our QA team and developers are still doing evaluations. If you have suggestions we’d love to get some in the comments below.

In Summary

You can see that our philosophical approach to all these areas is “just enough is enough” without having to buy in to an opinionated, vendor-locked or overly complex piece of technology. We’re staying low-level so that we can build out a coherent platform that suits our priorities and not spend a lot of time fighting frameworks. And there’s actually a very good reason for that.

Enough with the boilerplate!

So, you get it, we’re excited about what we can do and how we can do it in HTML and javascript. But here’s some cold hard truth: we lost important architectural patterns when making the move from Flex 4 and Swiz to Backbone. Namely:

  1. Data-binding
  2. An intelligent view component lifecycle
  3. Declarative views
  4. Event mediation

To get these back in you need to write a lot of boilerplate or use a more abstracted library on top of Backbone (which is fine). However, a couple of the newer frameworks attempt to implement items 1-3. Ember is actually designed around them and it feels like AngularJS and Knockout are in some part inspired by these features of Flex, but they are all pretty awkward in their own way. While Ember 1.0.PRE explicitly supports a declarative view hierarchy with {{outlets}} and the Objective-C style first responder system is nice, writing your own bindable EmberHandlebars helpers needs to be streamlined as does the syntax for declaring view actions. The overly complex interplay between the binding mechanisms and the models, collections and controllers make them time consuming to debug when they don’t play nice. The router is actually nice to write for though the heavy reliance on string literals for instance lookups and even for property sets/gets (which is an unnecessary recurring theme in JS frameworks) makes me sad as it’s more brittle. Angular, in its own way, is a lovely piece of engineering but wiring that much functionality into the DOM is bit adventurous, opaque and the project just wasn’t far enough long when we first started re-writing our apps. Knockout is in the same family.

We handle event mediation simply through an application event bus that enables modules like metrics and logging to peek at what’s happening in the running application without their code being strewn everywhere. It also allows modules to request the triggering of an action without needing to care about who actually performs it.

Regardless, we’ve been constantly impressed by all the thought and sweat that’s gone into each of the technologies we’ve talked about and as part of our maturation as a team and a company, we’d like to start to contributing back to this wonderful community. We’re currently pushing to get some of this foundational work on our roadmap. In the meantime, I guess we’ll keep writing our own boilerplate.

But wait! There’s more!

Projects We’re Super Excited About

  1. Meteor
  2. Derby
  3. AngularJS

If you’re excited about any real-time or other technologies that help create a fantastic user experience please let us know in the comments below, we like great technology and excitement.

Tech Shoutouts

Finally, we’d like to take some space to say thank you a few of the myriad other open-source projects and utilities that have allowed us to move as quickly as we have and helped us fall in love with the community. In case you haven’t heard of one of these, check them out, they might help you too. In no particular order: livestamp.js, lodash, slimScroll, momentjs, require-handlebars-plugin, express & jade, detectmobilebrowser, and of course, the jquery core team for doing such a good job pushing the edge for so long.

Leave a Reply

Your email address will not be published. Required fields are marked *

*

code

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>