I have read a few articles about the merits of decoupling a CMS framework from the web application, and most of these describe the theory rather than the practice. I decided to write this article in part to correct that and in part to describe our experience doing this with Drupal 8.

I-Next’s I-ntarsia platform is now built around Drupal 8, and is used by agencies, third parties and developers to reduce their costs in delivering their online projects. One long term user of our platform specialises in employee benefits services and uses I-ntarsia for systems such as HR portals (including SSO IdP), reward and recognition systems, long service awards and other custom sales incentives. The project described here is a corporate Christmas gift system for a customer of theirs who is a major UK media company.

Problem Description

So here’s the project in brief for a corporate Christmas gift system:

• 24K employees choosing a corporate gift in restricted timescale from a 400+ item catalogue
• 24x7 media business
• Employee data updated daily
• Over 60% of users access in the first 4 hours
• 50% of  ‘orders’ placed in first 4 days
• Accessed from highly restricted IP list through the corporate firewall
• SAML2 authentication from the corporate intranet
• A significant number of shared device users
• The client has outstanding internal network bandwidth
• 5-6 suppliers running various levels of ERP systems and requiring various levels of stock management and reporting
• SMS delivery of e-codes for some gifts
• Integrated call centre support system for any problem resolution
• Client security teams must sign off penetration and web application security tests

It is basically an e-commerce system, it’s just that nobody has to pay!

Previous Configuration

We’ve been lucky enough to have been asked to solve this problem each year for a few years now, and so we are familiar with the particular issues it throws up.

Our original approach to the problem was born in the days when browsers were one of the weakest links in the chain. Indeed, the original system had to support a large user base on IE6! So we took a tried and tested approach with functional demarcation based on resource requirement and front end vertical scaling. The solution looked something like this:

I-ntarsia 2.5 was our 100% proprietary CMS/Framework. The decoupled front-end and back-end were achieved using what, in I-ntarsia 2.5, was called our High Traffic Site Module. The module flattened content heavy on database activity but light on user state and passed that in a form requiring the minimal amount of database activity to a front end version of I-ntarsia 2.5 for production access. The frontend server, however, still served pages, ran the e-commerce database and operated as an autonomous website server. Images and the SAML session management process were split out to lighten the load on the front-end server.

There are many other variations on this that we could have utilised, but cost constraint and timescales prevented us from using many of them. Every pound spent on the system was a pound less spent on thanking the employees!

This setup, and variations thereof, have delivered successful programmes over a number of years, however, there are some serious disadvantages:

• Customer experience deteriorates with server load
• Peak loads can appear like a DoS attack
• All traffic hits the front-end server regardless of its ability to handle the requests
• Vertical scaling requires OS downtime to adjust, even on a cloud platform
• Managing changing load as the programme runs is cost inefficient, for most of the programme the front-end server has to be over-provisioned
• Content updates require the flattening process to run between the back-end and front-end

Despite these shortcomings, this system successfully delivered the service for a number of years.

Why Change?

A change would have been inevitable as the I-ntarsia platform has always been subject to a development cycle. However, two things opened up the opportunity for a complete re-architecting of the system:

1.  We migrated from I-ntarsia 2.5 to a Drupal 8 based I-ntarsia 8.x in 2016 as described here and here.
2. The client updated the corporate device standards and legacy browsers were removed from the user base

With these changes in place, we decided to rebuild the system to utilise new software and to address the shortcomings we had identified.

An Updated Approach

Our top issue with the previous architecture was the deterioration of customer experience with server load. Interacting with the site, flicking between gift categories and pages, slowed at peak times for all users and the only option for fixing that, having exhausted caching systems, was to scale the front-end server. With the restriction on legacy browsers lifted, we were able to consider using a 'single-page' javascript application to create a consistently fast customer experience.

In this approach, a single page is served to the user browser which contains a javascript application. Using its own data and data provided from other sources the application displays its state in the browser as a 'page'. When the user navigates using the menus displayed, they are mostly simply changing the state of the application, causing the application to swap between sets of data that it already has. As the vast majority of these are already available to the application in memory and do not need another round trip to the server, the transitions appear fast (almost instant), giving the user an excellent catalogue browsing experience.

Another benefit of adopting this approach is that we were able to construct a server and application to take advantage of asynchronous, non-blocking execution which was not available to us in the previous approach. The simplest analogy of this is the difference between calling someone to ask a question versus texting them to ask the same question. In a call you dial, you wait for them to answer, you ask the question, get the answer and then hang up. For the time between starting the call and hanging up you can do nothing else, you are 'blocked'. With a text, you send them the text and then get on with whatever else it is you are doing. When they respond to the text, you are notified that they have responded and you can then deal with that response.

Solution Framework

To build the solution we chose the following elements:

• Node.js as a front-end server (FES): Whilst there are plenty of notable alternatives, Node is simple to set up and run as an FES and could be deployed in our platform easily.
• Angular web application framework: Again, we could have chosen others but more members of the team had experience of Angular development.
• Drupal 8 backend: As our platform has Drupal 8 at its core, this is no surprise. However, this is an on merit selection, as will be detailed later in this piece.
• Request Tracker (RT) for call centre support: Flexibility, stability and performance make this a good choice, and topped with a capable REST interface, RT was a no-brainer for us.
• CI/CD processes managed through I-ntarsia GitLab service

2018 Configuration

The resulting configuration in 2018 looked like this:

 

The key benefits of this configuration were:

• Dynamic response to load with no downtime.
• Right-sizing servers. 
• Continuous integration/development with no downtime.
• Live catalogue updates.

The right-sizing of the servers was a key win for this system as the only functions that really need more than a default configuration (1 core and 1GB of memory) was the Drupal backend. Even then it had only an eighth of the resources of the equivalent in the previous configuration.

Drupal 8 Decoupled

The choice of a backend framework for this type of solution is critical to its success and we had the following criteria:

• Robust and powerful 
• Flexible architecture
• Easy to use content management
• Powerful REST API
• End-to-end security

Our platform is based on Drupal 8 but we were open to other options. In reality, however, Drupal 8 exceeded our expectations in our initial testing. Here's an overview of why:

• State of the art content management framework. The administration experience (AX) is as easy and powerful as it gets right now. By no means perfect, but certainly robust and powerful.
• Outstanding flexibility in information architecture (IA). Simply using core content, entity, taxonomy and view functionality we could build a custom IA without the need for any additional modules, not even a repurposed e-commerce module!
• Core REST capability enabled custom API development. The out-of-the-box capability of Drupal 8 is excellent. We supplemented this with the only third-party module we used which was the REST UI module, which was used for convenience and speed.
• Proven stability, security and scalability. If you have any doubts about Drupal's credentials on these, just hit Google.

Out-of-the-box we found Drupal 8 to be a highly capable backend server. We only needed to create two tiny modules for the entire backend part of the project to enable additional logging and user communication.

• Gift Reset Form: This is done via a request from the RT web GUI which only supports a GET request (without additional dev).
• Session Log: We track SSOs and needed to log some state information. A REST PATCH did not support what we required.

Total code for the backend server, including module set up, is under 500 lines! 

System Performance

The system launched in late November and allowed orders until the 28th December and then ran in support mode for a further few weeks to help clear up delivery and other gift-related issues. The new configuration allowed us to dynamically manage load, particularly in the first few hours and days. The key performance highlights were:

• 30K SAML2 logins in the first 24hrs
• 14.5K unique users browsing the catalogue in the first 24hrs
• 8 orders per second at peak
• 3.8K orders in the first 24Hrs (35% increase)
• 0% unavailability
• 0 performance issues raised by the audience
• 33% reduction in call centre reports

To put that into some perspective, if an e-commerce company could generate an annual equivalent demand for products they would be receiving 26K orders per week or 1.4M per year. Apply even a modest average order value/margin and you have a highly cost-effective system.

Outcomes

The outcomes of this project were as follows:

• A happy end client, always the number one consideration.
• System users experienced 'one-to-one' performance rendering all of the technology 'invisible'.
• We added a rollout of standard Node based front end servers to the I-ntarsia platform.
• I-ntarsia's integrated Gitlab service proved itself on a 'hot' project.
• The small development team learned a great deal.

Conclusion and Thanks

When you embark upon a project such as this, you expect a significant amount of software development in the Angular application. You also expect to be writing code or testing custom configurations of the backend to support your development sprints. What was truly amazing in such a complex project was how little of that was needed when we used Drupal 8.

Drupal's founder and lead developer, Dries Buytaert recently published this article about the various strategies for How to decouple Drupal in 2019. This would have been a useful reference when our team started work last year, we could then have clearly stated that we were building a fully decoupled Drupal application.

The conclusion of Dries' article is:

Drupal continues to be an ideal choice for decoupled CMS architectures, and it is only getting better

and I hope that this article demonstrates the practical reality of that statement. 

Rather than just leave it there, I'd like to thank and congratulate Dries Buytaert and the Drupal Community for the outstanding product that Drupal 8 is.