Striking a Balance Between Moving Fast and Incurring Technical Debt

I had a great time as an intern here at Personal Capital. I worked as a member of Personal Capital’s Data Team, getting my hands dirty in a range of projects, from data validation, to daily monitoring, to expanding/updating our ETL process into Amazon Redshift. One of my biggest takeaways was the paramount importance of not allowing the fast pace of the company to erode the use of best practices. Specifically, I grew to appreciate how important it is to write solid tests of all new code, keep good documentation of all changes to the database, and always keep security in the back of your mind.

 

Testing

 

The Personal Capital codebase is primarily written in Java, allowing us to write solid unit tests through a Spring framework. We hold ourselves to a high standard, mandating that our unit tests must touch a significant portion of all new code.  One of the larger tests that I wrote used Mockito to mock large portions of data through which to test our transformation, allowing me to better focus the test as well as maximize its robustness. This test operated on most of the tables in our most important schemas in Redshift, and I wrote it in such a way that it would be as easy as possible to add new tables to the test/amend it to accommodate changes to our Redshift schemas. Coupled with extensive logging, these tests allow us to pinpoint bugs when they are introduced into the codebase.

 

Documentation

 

During my time here, I was part of an ongoing effort to create a general-purpose data warehouse on Amazon Redshift, a single source on which all data exists, thus greatly improving the ease of analysis. By consolidating our data on Redshift, we can now run queries in seconds, queries that would previously take hours. As one might expect, throughout the process, it’s typical for fields to become deprecated, and for fields to be created with ambiguous meaning. Part of my job at Personal Capital was aimed at addressing this problem, through a unified data dictionary that resolves discrepancies in the definitions understood by different teams, as well as consolidates the documentation. This is still a work in progress, but it will ultimately make analysis of Personal Capital’s data much more meaningful. In addition, throughout my time here, I did work that involved deprecating/adding many new fields in existing tables, as well as adding new tables, and I made sure to bolster Personal Capital’s documentation by documenting my changes thoroughly.

 

Security

 

I was really impressed by the security measures put in place at Personal Capital. Shortly after I began working here, I was given a rundown of the security practices used by Personal Capital as well as information about how to avoid introducing vulnerabilities in the code. The engineers here always keep security in mind when writing new code, something that can be easy to forget in the effort to move quickly. In addition, all new code goes through a rigorous review process before release.

 

Conclusion

 

I’m very grateful to have been given the opportunity to work at a startup quickly establishing itself in the financial technology sector. While it is a very fast-moving company, I’m glad that the engineers here don’t lose sight of the long-term picture, always meticulously subscribing to best practices that will greatly improve our outlook years down the road.