Rebuilding a Monolith

It's no secret that microservices are the hotness right now (I won't say new at this point because they're fairly well established). But you only have to Google the phrase "Why not to use Microservices" to see a treasure trove of articles about why to be sceptical of this pattern. As with all things, in some instances it's the "correct" approach and in other's, it's not. I'm not entering into a holy war of which is better, nor am I a Basecamp hipster who despises frameworks. This is a story about why and how we collapsed a few "microservices" into our central API.

The Setup

Koru is a pre-interview assessment filter. That sounds wordy but basically it's designed for large corporations who get thousands of applications to jobs, and this tool aids companies in seeing what different strengths a candidate has based on an assessment. The product is fairly simple infrastructure wise. An event-driven microservices architecture with a central API, a React frontend for customers and a admin interface for employees. In addition to that we have systems to generate PDF's and send emails that are part of the candidate lifecycle through our system.

What we wanted to combine was the following:

It's worth noting that these were all separate repositories.

Why?

The reasons for doing this were simple

How we did it

As all things should be, we tackled these one at a time and released each one of them so we didn't have any confused boundaries about where large swaths of code got introduced.

  1. The documentation The documentation site was written using Slate, which is a Ruby based project. To add this to our NodeJS/Express API, sounds painful but all we did was put it in new folder 'devdocs' and treat it as it's own separate project. Next we added a GitHub Action (Our CI/CD tool of choice) to build and deploy the 'devdocs' to the S3 bucket where they are served from.

That was easy

Well that was easy

Right, next... 2. Internal Tooling API Since this was a new service written only a few months prior, it was already similar to our main API in terms of the patterns used. Additionally, it was written in Typescript so the code could quite literally be copied across. We took the time to fix up some tools that were broken and created new ones. Additionally, we added a few simple sanity tests for each one so that we could verify they worked.

  1. Admin Portal This is where things became a bit tricky. As stated previously, one of the main reasons we wanted to merge these API's was because the current API didn't really serve the needs of the system, we were sort of bending it to our needs and in a number of cases, it snapped under the pressure. We embarked first on a mission of discovery, what were all the things that it did, why did it do them, and how did we want to improve them.

The API was written in Python using Flask as this is what the previous team had mostly used. But since there was little expertise within the team for Python, we decided to rewrite it in Typescript. One of the main improvements we wanted to make was to merge the creation of different records in various tables into one easy to use endpoint. Previously, each individual section when creating anything (like a study, survey etc.) had to be saved individually. This lead to confusion from users and confusion when debugging issues where some data was relied on by another part of the system. By combining them, we made the save process one simple API call, and more importantly, one set of errors to deal with. Along with this merge, we added new tests, both unit and integration so that we could verify the integrity of this process. We also made plans to improve the service further by adding tools for Koru employees to be able to "Copy" existing studies to make them easier to setup.

The difficult part with this merge was authentication. In our main API we had no mechanism to authenticate requests only from Koru Employees. Thankfully this was a breeze, because we use Auth0 to handle authentication. We simply wrote a new middleware handler that reads from Auth0 based on the JWT req.user.sub field and checks certain properties of that user in Auth0's system.

The interface was built in React but although not inherently bad, it was clearly coded in a bit of a rush. Things over time had been hacked in as new features came along so we took the time to clean this up (as the boy scouts that we are) before merging it into our central tooling interface. Again we added tests to make sure that the interface loaded the data correctly and behaved the way we expected.

Conclusion

Overall, I'm really happy with "the big merge", because it's meant that we've created a useful resource for Koru employees and they can self-serve for whatever customers ask of them. Additionally, it will be easier to add documentation for the API and consolidate that back to a single work ticket. On top of this, the main Koru API is now a complete CRUD API rather than having bits of "R" and "U" with "C" and "D" being all the way over in a different service! Hooray!