Overview of every element of our stack: what it is, what's working, what's not.
Explanation of grade:
A - excellent. does what we need well and should last us for awhile.
B - good. does what we need, and generally good, but could use minor improvements.
C - ok. does what we need, but doesn't really do us any favors either.
D - poor. hasn't hurt us yet, but probably a disaster waiting to happen. needs work.
F - fail. this is actively holding us back.
Our server, Turtleboat, is responsible for collecting and storing financial data, performing calculations on the data, and serving it up in a REST API. It is based on Python/Django.
Our crawlers aggregate financial information from several sites. They are written in python, and occasionally make use of PhantomJS.
Our main DB stores and serves financial and user data. We built it on SQLite for its simplicity, and so we could easily share
.db files during early development, though now it is holding us back. Wrapping the DB is a Django object model (ORM), which is clean and does most of the work for the API and Admin, both described below.
Our search DB mirrors the main DB but provides highly configurable text search and scoring. It is based on ElasticSearch.
Our data is served up as JSON by the API, a REST API based on Django's Tastypie library.
Our app server is uwsgi, and our web server/cache is nginx. The app server is responsible for running the python app. The web server is responible for serving static files, proxying to the app server, and sometimes caching the results.
Our daily calculations analyze the financial data we crawl and store cached statistics on risk, return, expenses, and limited topical data.
Our admin tools are web UIs for inspecting and curating the data, semi-automatically generaged by Django.
We write unit tests for all of our tools and systems. These tests are run automatically every time code is pushed to the
We also write data tests — "validations" that test the consistency of our data. These tests run twice per day.
Our client, SM, is our first client app for the Turtleboat service. It loads securities information from the Turtleboat service and allows users to view and interactively manipulate portfolios. SM is a "thick" client, meaning that it contains all UI and behavior built in, and only needs data to run. It is built using Backbone/Bootstrap.
The models are the data loaded from the server, stored to make it easy to render the user interface.
Our HTML/CSS is beautiful on the inside and outside. We use the less preprocessor for CSS to keep things clean. Most of the UI is built out of
underscore templates which render data into the HTML.
Our charts are based on d3.js. d3 is an extremely flexible toolkit for fully-customized, animated visualizations.
We use custom widgets such as selection boxes and carousels, mostly provided by the Bootstrap library.
We use Google Analytics to track in-app pageviews and behaviors. We instrument most every event in the app, whether or not we use the analytics.
Our Ops are built on Jenkins and Amazon EC2.
We currently run our server on a single M1.Small instance. This keeps our costs down, and also forces us to keep our code efficient. It is easy to scale up to much larger instances later if we need.
We deploy using Jenkins jobs that run shell scripts which copy files and configurations to NGINX virtual servers. Every push to
develop branch triggers a fresh deploy to
staging servers. Deployments from
master branch are done manually with the push of a button.
Daily calcs are run by Jenkins once per day.
Tests are run by Jenkins. They are run on every deployment. Data validations are run twice per day on a timer.