The tech stack part 1

001-01Family-Tree

 

At Personal Capital creating better financial lives through technology and people.

I have just joined the front end team and I’m loving it. The way we live in a startup pace, moving agile and collaborating every cycle surrounded by amazing people makes it perfect for professional development.

We have a great tech/skills stack and I wanted to share it with everybody especially those who are thinking in joining our team.

We believe in the power of technology to change the financial industry, making it more accessible, affordable, and honest. And we believe in the power of people to change the nature of investment advice, making it more transparent, objective, and personal.

Living in a world that is constantly talking about saving money and figuring out complex investment decisions, our front end engineers face a huge challenge: to make things as simple as one click.

To make that possible, we process lots of data. We utilize user information to generate what we call a progressive profile. Every user has a progressive profile which is generated from their individual inputs.

By doing this we can recommend plans, proposals and functionalities so our users can get the most out of our platform. This is how our award-winning platform allows 1.2 million users to track and invest their assets.

Ensuring that our technology operates on multiple devices is a must, not only a web platform, but within easy-to-use apps. Our users want access to their finances at home and on the go and that is what we deliver.

The Front-End Stack

This article will be cover our web platform tech stack, the one that we use day to day to make things as easy as one click/tap.

On development and build time we use Node.js, which we mix with some browser-sync flavor to proxy all non-static requests to a development server in the Amazon Cloud, Node and browser-sync serve the static resources to keep it simple. The team specifically chose not to use any build system like Gulp or Grunt to avoid having extra external dependencies.

Since the front-end layer is simple, NPM manages all the tasks.

Single Page Application

Our back-end services are written in Java, and we consume them using Backbone and Angular. This allows us to reuse tons of pre-implemented open source plug-ins and create reusable components that can be implemented across all of Personal Capital application’s domain.

Html

Most of our platform is written using handlebars templates. This gives us the unique ability to use lots of pre-build helpers and accelerating the prototyping and development process.

Styles

We are migrating from Bootstrap to Inuit, refactoring code and creating our own styling framework based in OOCSS/BEM. By doing this we can ensure coding style consistency across our styles sheets, generating maintainable beautiful code that is reusable and easy to understand.

This is especially important since the company is growing rapidly, and by doing this we help our new engineers to jump in the train faster and deliver value sooner.

Data Visualization

If you use our platform, you have seen a lot of beautiful visual components to our design. We use graphics whenever possible to make complex data easier to understand.

Our graphics manage complex datasets. They are dynamic so you can interact with them and get from the most generic information to the more specific transactional details.

To achieve this, we used Raphael. But as it is no longer being maintained, we are transitioning to D3 library, which is much more flexible and powerful, giving us the tools to generate almost any kind of graphics that we can think of.

Testing

The front-end team uses Mocha/Chai/Karma for writing unit tests. The QA team uses Selenium for automation. When we picked those frameworks and tools, we evaluated them thoroughly to fit our needs.

Mocha, along with Chai (expect/should – BDD style) worked out well for us with a effective yet readable tests. I cannot emphasize the importance of readable tests enough. Now the feature set can be seen across the organization, comparing the expected outcome with the outcome. We’lI have to do a write up sharing more of our excitement.

The use of Mocha and Karma helped us on the path to continuous delivery and immediate regression reports. Tests are executed periodically by our continuous integration server, which monitors the health, watches for the codebase consistency and gives us the code coverage information. This is really important to ensure that we’re delivering a first class service/product.

Back in 2014 we wrote an article about this, you can find it here.

Automation and CI

We use Jenkins as the orchestrator to build, test and deploy all of our environments. Our DevOps team wrote the tasks that run automatically or on-demand with one click.

Jenkins triggers Karma to run our Mocha test and Selenium to run-regression. Those tests are run in different browsers, the ones with head and without it (AKA headless browsers).

BTW, Karma watches our files and runs tests in development time too, eliminating the necessity of doing so before pushing code and giving us immediate feedback on what we are coding.

We also have integration with Husky that prevents our dev team from pushing code up to the origin/remote if there are failing unit tests.

Versioning

We use Git and we love it. The workflow is pretty similar to git-flow. We just extend it, adding a few extra branches like a sandbox one, where we make al kinds of experiments.

Once we run “git push”, the linters and code style police go through our code to ensure consistency and good practices. This wasn’t the same before, which was a big problem for our engineers due to JavaScript flexibility. Now the codebase keeps getting consistency across the apps.

Agile

At Personal Capital, everyone is a team player. We volunteer to get tasks assigned based on our expertise. We use scrum across all of the engineering teams and our sprints last one extreme week.

Automating your javascript unit tests – Karma

why

  • instant feedback while writing your code and eliminate the need to remember to run tests before checking in your code. thus leading to stable builds.
  • continuos integration with our hudson build system.
  • testing on real and multiple browsers.

what

  • set up karma to watch your source files and run tests on code changes.
  • set up karma to run the tests during hudson build and validate the build.
  • set up karma to track our code coverage.

how

We use mocha as our test framework. This along with chai (expect/should – BDD style) worked out great for us with an effective yet readable tests. I cannot emphasize the importance of readable tests enough. We had a team member who did a feature walk through by running through the tests which i thought was pretty rad. Product and QA could easily see what was the feature set, what was expected of and what was the outcome. I guess we have to do a write up sharing more of our excitement.

Before karma, we were running tests using individual test files. More often, you are working on multiple files and remembering to run tests on all these files manually was becoming cumbersome and error prone. So we started researching on test runners and karma seemed to fit all our necessities: automation, continuos integration, run tests on multiple real browsers and support for mocha.

set up karma to watch your source files and run tests on code changes

This was fairly straight forward. Karma’s setup is driven by a single configuration file where in you provide the location of files you want to watch for changes, browsers that you want to run tests, your testing framework and any preprocessors. Here’s a gist of our configuration file. The only tricky part was preprocessors. We use handlebars along with requirejs-handlebars-plugin for our templating purposes and serve our templates as individual html files. This was causing a problem karma was converting them into js strings because of its default preprocessor: html2js. It needed a bit of reading, but the fix was simple enough. The following additions to the config file fixed the problem.

preprocessors : [{'scripts/**/*.html' : ''}]
files:[...{pattern: 'scripts/**/*.html', served: true, included: false}]

set up karma to run the tests during hudson build and validate the build

We created another karma configuration file for this purpose. We added a junitReporter  so that we could export the tests in a format that could be interpreted by our hudson setup. The key differences are as follows. We are currently using phantomJS for testing on our build systems, but in near future, we want to extend this to real browsers.

reporters: ['progress', 'junit']
junitReporter: {outputFile: "testReports/unit-tests.xml"}
autoWatch: false
browsers: ['PhantomJS']
singleRun: true

set up karma to track our code coverage

Once we were able to configure karma to run in hudson, this was just a natural addition. The only additions to the karma configuration are as follows.

reporters: ['progress', 'junit', 'coverage']
coverageReporter: {
 type : 'cobertura',
 dir : 'coverage/'
}
preprocessors : {
 '**/scripts/**/*.js': 'coverage'
}

As you may have noticed, i may used simple and straight-forward words quite a few times and that is what karmajs is all about.

reads

http://karma-runner.github.io/0.10/index.html

http://googletesting.blogspot.com/2012/11/testacular-spectacular-test-runner-for.html

 

https://www.npmjs.org/package/karma-handlebars-preprocessor