2013 HTML5 Dev Conf Key Takeaways

 
  • Embedding complex SVGs into HTML
    • great talk about SVG and how box.js has used it to make its amazing viewer. apparently its more performant than the in-house chrome pdf viewer.
    • highly recommended read.
  • Scale and Performance: Data visualizations in modern browser
    • while the earlier talk was about not using canvas, this was all about using canvas and its might.
    • very inspiring to what folks have done and seems to be most of them are in publishing business (NYT and various media companies)
    • this and couple other similar talks strongly advocate use of D3.js for visualizations
    • Learned about Datawrapper, a tool to create simple, embeddable and interactive charts.
  • Constraint validation: Native client side validation web forms
    • talks about native validation that is available and supported in most browsers
  • Transforming the presentation of official statistics
    • amazing infographics and very inspiring to see huge amounts of data represented in easy to understand, interactive graphics.
  • ReactJS
    • new mv* js library from Facebook … oh what !!
    • its pretty much against everything that the current set of mv* libraries advocates
    • no templates, re-rendering your whole application when data changes… what??
    • and that is why it is a interesting read…never listen to anybody that says “DONT RE-INVENT THE WHEEL”
  • JS Inconsistencies Across Browsers
    • very illuminating and gives you a brief insight into how browsers work and why the inconsistencies.
    • talk about event loop, iframes and xhr gives insight into how browser processes them and advocates good coding practices when using them.
  • Continuous Delivery for JS Apps
    • the highlight of the conference for me. great speaker and very motivating
    • key takeaway: AUTOMATION

Distributing Static Content From CDN

why

  • Distribute content to end users with low latency and faster response times.
  • Ability to deploy new static content without having to re-deploy the entire application.

what

  • Identify a Content Distribution Network (CDN) solution that meets our needs
  • Setup origin server(s) to store the content that will be distributed via CDN.
  • Setup a distribution to register the origin server(s).
  • Reference all static content in the application with the distribution’s url.
  • Version static content to bust the cache both in the browser and CDN.

how

Before we get into implementation details, what do we mean by “static content”?  Here, we define static content as all the web resources that are not specific to a particular user’s session.  In our case, this includes css files, javascript files and all media files.

Identify CDN solution

Since most of our infrastructure is hosted in Amazon Web Services, the AWS CloudFront CDN was a logical default choice to try.  Like most of the AWS application-level services, CloudFront tries to hit a reasonable balance of the 80/20 rule:  it is low-cost, easy to operate, and covers most basic functionality that you want in a application service, but does not offer much in the way of high-end differentiating features.

Setup origin server(s) to store the content that will be distributed via CDN.

An origin server in CloudFront can be either a web server, or, as in our case, an AWS S3 bucket. We have setup an origin for each of our environments (dev, qa, prod). We then have a Hudson job (build) for each of our environments. The Hudson job checks out the corresponding branch from our git repo for the static content; processes it as mentioned in this post; updates a version.xml file with the Hudson build number; zips the file; and copies it to a S3 bucket. This bucket, however, is not the CloudFront origin. It is like a docking area to store all zipped build files from our various Hudson jobs. We have another Hudson job (deploy) that copies a given zipped build file to an origin bucket. More on that in a minute.

We have a different Hudson job for each environment because the static content could be processed differently based on an environment. For example: in our dev environment, we do not combine or minify our js files. In our qa environment, we combine but do not minify our js files. In prod, we combine and minify our js files.

Back to the Hudson deploy job mentioned above. This job takes two input params: the name of the zipped build file to be deployed and the environment to be deployed. It simply unzips the build file into a temporary directory and uses s3sync to upload the content to the appropriate S3 origin bucket for the given environment.  And from there, it is available for distribution via CloudFront.

In addition, in our dev environments, we use a Continuous Integration (CI) process, where our Hudson jobs send SNS messages to our the web server when a build is available.  The web servers pulls the static content build from the S3 staging bucket and then serves it directly from Apache.  This allows for a more targeted integration test of the static content without bringing the CDN mechanisms into the mix.

Setup a CloudFront distribution to register the origin server(s).

We have a CloudFront distribution for each of our origins.  A CloudFront distribution associates a URL with one (or more) origin servers.  Creating the distribution is easy via the AWS Console.  In our distribution, we force HTTPS access – since our containing pages are served via https, we want to ensure the embedded static content is as well, to avoid browser security warnings.

CloudFront does allow you to associate a DNS CNAME with your distribution, so that you can associate your own subdomain (like static.personalcapital.com) with your CloudFront distribution.  This is more user-friendly than the default generated CloudFront domain names, which are like d1q4amq3lgzrzf.cloudfront.net. However, one gotcha is that CloudFront does not allow you to upload your own SSL certificate.   So, you have to choose between either having your own subdomain or having https – you can’t have both (some other CDN services do allow you to load your own SSL certs).   In our case, we chose https, and use the default cloudfront.net URL.

Reference all static content in the application with the distribution’s URL.

We have an environment config property that stores the corresponding CloudFront distribution’s URL. This property is available for all server-side webpages where it is referenced as follows:

 <link rel="stylesheet" type="text/css" href="<%=staticUrl%>/static/styles/css/main.css">

We then needed to make sure that all our static content references its resources via relative URLs to keep them independent of the distribution URL. For example, an image reference in main.css would be as follows:

 background: url('/static/img/dashboard/zeroState.png')

which would resolve to the root URL of main.css which is the distribution URL. I would like to know if there is a better way to solve this.

All our javascript uses relative paths anyway because of “requirejs” so we did not have to make any changes there.

All other references to static resources were on the server side where they had the config property to correctly reference the static resource.

Version static content to bust the cache both in the browser and CDN.

All our static content have aggressive cache headers. Once a static resource is fetched by the browser from the server, all future requests to that resource will be fetched from browser’s cache. This is great but when a newer version of this resource is available in server, the browser won’t no know about it, until its cache entry for that resource expires.

To prevent this, we use a common technique called URL fingerprinting wherein we add a unique fingerprint to the filename of that resource and change all its references in the webpages (JSPs) with the new filename. The browser, as it renders the updated webpage, will now request the resource from the server since it treats it as an entirely a new resource because of its new name.

The Hudson build job mentioned above processes our static resources, versions them with the build number and also stores the build number in version.xml. The version.xml file is then used by the application to retrieve the version number and pass it onto web pages at run-time. This helps us achieve our second goal that of keeping our static (front-end) development independent from our sever (back-end) development. This is very powerful as it gives us the ability to change our static content any time, have it deployed to production and not worry about updating server webpages with the latest version number. Pretty neat ah !!

Versioning of the resources also helped us out a great deal with our CloudFront distribution. CloudFront distribution behaves very similar to how browser handles resource caching. It does not fetch a newer version of the resource from the origin server unless one invalidates the current resource in the distribution. This has to be done per resource and it has a cost too.  The CloudFront documentation offers more considerations regarding invalidation vs versioning.

There is one other workaround you could use to force CloudFront distribution to fetch the content from its origin server. Set the distribution to consider query strings in the resource urls. And then pass a unique querystring along with the resource url and it will force the distribution to fetch the resource from the origin server.

That is it !!

reads

Bundling Front-end Resources with RequireJS

why

  • faster load times because of fewer http calls and lesser data over the wire.
  • effectively invalidate browser cache when the newer version of a resource is available.
  • lends itself to manage these resources via CDN

what

  • combines and minifies javascript
  • combines and minifies css
  • revises file names of js/css files to support heavy browser caching
  • updates the html to reference these new hyper-optimized js/css files

how

Lets start with combining javascript. This could be as trivial as writing a shell script to concatenate all of your script files into one single file. Works great for monolithic applications but for large-scale javascript application where significant amount of data manipulation and display is done at the browser level, having a modular architecture greatly helps.

Modules are an integral piece of any robust application’s architecture and typically help in keeping the units of code for a project both cleanly separated and organized. They need to be highly decoupled and represent distinct pieces of functionality thus facilatiting easier maintainibility and easily replaceable without affecting the entire system.

“The secret to building large apps is never build large apps. Break your applications into small pieces. Then, assemble those testable, bite-sized pieces into your big application” -Justin Meyer, author JavaScriptMVC

However, there are two main problems in writing modular javascript:

  • No means to import modules of code in a clean and organized manner.
    • Developers at present use variations of the module or object literal patterns to write modules. The modules are exported as namespaces of a single global object where it is possible to incur name collisions as the application and team grows.
  • No cleaner way to handle dependency management.
    • Developers use the exported namespace to include the modules. And then load the corresponding script files in the order they are included. This becomes really difficult as the application grows and particularly when scripts start to have overlapping and nesting dependecies.

There are various script loaders that help out but again there are no native means that solve the two problems in a holistic manner. And this where Asynchronous Module Definition (AMD) greatly shines. It has greatly helped out in architecting our application, the benefits of which will be a different post, but the following pretext from Addy Osmani should summarize well its use.

The AMD module format itself is a proposal for defining modules where both the module and dependencies can be asynchronously loaded. It has a number of distinct advantages including being both asynchronous and highly flexible by nature which removes the tight coupling one might commonly find between code and module identity. Many developers enjoy using it and one could consider it a reliable stepping stone towards the module system proposed for ES Harmony. – Addy Osmani

The following example demonstrates how we address the first problem with AMD:

define(
 	module_id /*optional*/,
 	['dog', 'actions'] /*other module dependencies if any*/,
 	function( dog, actions){ /*function for instantiating the module or object*/ 
 		//create your module here
 		var myModule = { 
 			dogActions: function(){
 				console.log('dog actions');
 			}
 		} 
 	//return a value that defines the module export
 	return myModule; 
 });

We have solved the second problem by using RequireJS, a popular script loader and we felt it is the natural fit for AMD. It uses the AMD module specification for defining and requiring modules, and it loads these modules via a built in script loader. It also provides for loading non-script dependencies such as text files and we have used that ability to load our handlebar templates.

Lets extend the above example to demonstrate how well AMD and requireJS compliment each other to keep your code modular, yet bought together without one worrying about module dependencies and name space conflicts.

<script src="scripts/require.js" data-main="scripts/main"></script>
// In scripts/main.js: your application bootstrap
require( 
	['dog', 'actions', 'hbs!templates/dogActions'],
	function( Dog, Actions, DogActionsTemplate ){
 		document.body.innerHTML = DogActionsTemplate({
 			dog: Dog,
 		 	actions: Actions
 		});
 	}
);
// In scripts/dog.js
define([], function( ){
	var dog = {
 		name: 'lula'
 	};
	return dog;
});
// In scripts/actions.js
define([], function( ){
	var actions = {
		name: 'fetchBall'
	};
 	return actions;
});
// In scripts/templates/dogActions
<h1>{{dog.name}}</h1>
<ul>
<ul>{{#each actions}}
	<li>{{this}}</li>
</ul>
</ul>
{{/each}}

In addition, RequireJS also provides an optimizer tool that we could use to build our entire javascript into a single file or two for production. And thus, how we address combining javascript.

Combining css was easy. We use sass to write our style sheets. As the name itself suggests they are awesome and were introduced to me by awesome fellow developer, Justin. We use scout or similar tools to output a single css file from our sass files that acts as the stylesheet for the entire application.

Now that we have successfully combined javascript and css files, lets compress them. We have earlier mentioned script loaders and how require’s optimizer tool helps out. Currently require provides two options: uglifyjs (default) and closure compiler to minify. we use uglifyjs and have it configured to “beautify” in our dev environment so that we could still effectively debug. Source maps seems to be an interesting concept in this arena that would help you debug “production” scripts.

Now that we have combined, minified our resources, our next step is to add a build fingerprint to the resource file names so that the browser treats them as new resources and invalidates its cached version of the resource.

We have used bunch of maven-ant scripts to achieve this. More on this in my next post.

challenges

One of the major challenges with requireJS is using non-AMD compatible scripts, mainly the third-party libraries. When we initially started with requireJS, we simply got the AMD versions of the third party libraries we needed or modified the libraries to be of AMD format. But we quickly realized that is not a feasible approach and was a major pain point when we had to add a new library. Tim Branyen had a good blog post on how to overcome this problem which was later incorporated into requireJS 2.0. With the introduction of shim, this problem was easily addressed in the config file as follows:

require.config({
 shim: {
 'backbone': {
 deps: ['underscore', 'jquery']
 , exports: 'Backbone'
 }
 , 'underscore': {
 exports: '_'
 }
 , 'raphael': {
 exports: 'Raphael'
 }
 }
 , hbs: {
 templateExtension: 'html'
 , disableI18n: true
 , helperPathCallback: function(name){
 return 'templates/helpers/' + name;
 }
 }
 , paths: {
 jquery: 'libs/vendor/jquery-1.7.2'
 , underscore: 'libs/vendor/underscore'
 , Handlebars: 'libs/vendor/handlebars_hbs'
 , raphael: 'libs/vendor/raphael'
 , backbone: 'libs/vendor/backbone'
 , hbs: 'libs/vendor/hbs'
 }
});

 

reads

http://addyosmani.github.com/backbone-fundamentals
https://github.com/h5bp/
http://www.nczonline.net/blog/2010/07/06/data-uris-make-css-sprites-obsolete/
http://addyosmani.com/blog/yeoman-at-your-service/
http://www.youtube.com/watch?v=Mk-tFn2Ix6g
http://www.html5rocks.com/en/tutorials/developertools/sourcemaps/