Haskell in Production: Bdellium

I was reading the comments on a Haskell post on Lobste.rs and someone said that while there’s a decent amount of posts around Haskell, there aren’t enough about people using “Haskell in Anger/Prod”. I agree.

I’ve been fortunate enough to work for two different companies that are using Haskell in production successfully and I thought I’d tell the story of one of them.

Bdellium is a small company in the financial services sector providing analytics and reporting tools mainly to the US retirement industry. When I joined the company as a full-time employee I was the only programmer in the company and I inherited a relatively small Haskell application for doing retirement plan analytics and optimization. At this stage it was more of a proof of concept than a product and we paid our salaries largely through existing consulting contracts. We were not focused on writing “beautiful code” or interested in solving challenges that weren’t real to our business, we wanted to build a basic, reliable product that could quickly generate revenue to fund further development.

The first prototype of the product, before any code was written, was made in Excel by the CEO of the company. The developer who was working on turning this Excel prototype into a working proof of concept in real code then did his research, prototyped in Python, then Clojure and finally in Haskell. Haskell was an immediate fit for the problem domain. Not only does Excel map quite well to immutable data structures but the level of safety and guarantees that the language and compiler gave us was a clear win in an environment that was largely about numerical algorithms, mathematical formulas and data transformation.

When I came to the scene I didn’t have much Haskell experience, but I was feeling productive and contributing significant functionality to this prototype after about 2 weeks. I had programmed largely in Ruby on Rails for about 6 years prior to taking over this codebase. While some of the ease of starting to work with this codebase could be attributed to the talented developer that preceded me, I have taken over Ruby codebases of similar size from some of the most talented Ruby programmers there are and have always had a much harder time than I had with this codebase. Haskell let me quickly browse the code, read the types and almost instantly understand the structure and layout of the program. It felt as though the compiler and the types were actively directing me to where I could find the things I was looking for.

I continued developing this codebase into a saleable product/service over the next year or so. This was essentially a year of firefighting, constantly adding features and functionality and very little time to refactor or reconsider past architecture decisions. Combined with the fact that I wasn’t a great Haskell programmer when starting out, when we launched the product and started making money from it, the codebase had grown convoluted and wasn’t in a good state to move forward. With the money made from the first customers of the product we hired another developer, who came largely from an R background. He sat down and within one month had refactored most of the codebase into a clean, well-structured application that was just as productive to work in as the prototype application I inherited. If this application had been written in Python or Ruby, the first thing on my mind when looking at it would have been to rewrite it from scratch. Because of the ease of refactoring in Haskell, partly thanks to the type system but largely thanks to having almost only pure functions, refactoring was not only possible, but so easy that someone who wasn’t familiar with the codebase could do it in a matter of a few months, with zero service disruption.

At the start of the products’ life we mostly analyzed small retirement plans with 100 to 500 plan participants. As time went on we started seeing plans with 2,000 participants and even 10,000 participants. At these numbers the execution time for the processing started going outside of acceptable bounds. Luckily, every participant could be analyzed in isolation from the others and so the problem was embarrassingly parallel.

I changed one line of code from

map outputParticipant parts

to

map outputParticipant parts `using` parListChunk 10 rdeepseq

and execution times were now about 3.7x faster on our 4-core server. That was enough to entirely fix the issue for another 6 months, during which time we could focus on further enhancing the product instead of worrying about performance. Later, we did do extensive profiling and optimization of the code to further reduce execution times significantly, but we didn’t want to prematurely optimize anything.

The second hire we did we posted the job publicly on Reddit. I’ve had experience hiring Rails developers before, but this was an entirely different world. Posting a Rails job usually leads to a few hundred applications going through the major boards and roughly 10–20% I’d say are people that I’d actually be interested in hiring. While I only got about 50 responses to the Reddit post, all of them were people that I could have hired. Many of the applicants were even grossly over-qualified and were willing to take the job just because it would let them work in Haskell.

Ultimately I do not believe we could have accomplished what we did in those early days without writing our analytics code in Haskell. The productivity and level of correctness that the language offered was what made it possible to write the application in the amount time we did. During development and more importantly during years of running in production, we saw little to no bugs, simply by avoiding partial functions and encoding as much as possible into the types.

Bdellium has now successfully launched its second product, also written in Haskell, and is gaining traction faster than ever before.

Thank you for reading! If this post was interesting to you and you’d like to hear more “Haskell in Production” stories, please let me know. I also encourage all the other Haskellers out there to speak up about how and why you’re using Haskell!