The Headless Puzzle
Solving the Headless Commerce and Content Puzzle
The business case for the move to a headless commerce and content platform, whether that’s making use of the new breed of SaaS applications which make up space previously filled by the so-called ‘monoliths’ (commerce, search, content management, OMS, CRM etc), building your own microservices or a mix and match approach, is increasingly well-articulated. However, the often-overlooked factor in this journey is the significant operational changes the transition will necessitate if you are to achieve a manageable, scalable, secure enterprise platform going forward.
There are quite literally hundreds of variables and questions that will need to be addressed. This complexity will take a considerable amount of time and resources to carry out efficiently and every business tends to address these factors differently even though the desired outcomes are remarkably similar.
Up to the onset of lockdown, E2X had been hosting the London based Headless Commerce meetup for over a year. This provided the opportunity to discuss the subject with many of our peers, from brands and businesses in multiple verticals to well-funded tech-savvy startups and some of the world's most recognisable brands. We have reflected on our own experiences, addressing operational challenges with clients who we have supported in the transition to ‘headless and microservices’ in various ways, for over the best part of ten years.
So, what are the lessons learned and what advice would we give to others embarking on this journey; or who are already on the road but are finding it trickier than they first imagined?
This article is the first of two parts that will explore these questions by referring to several enterprise personas that, while being completely anonymised, reflect the types of businesses we have had the opportunity to work through these issues with. Both in our client projects and the experiences shared amongst the community involved in the London Headless Commerce meetup.
So, how can we usefully reflect on the lessons that could be learned by different business ‘types’ that are travelling this road? The starting point should be considering what key characteristics make up these enterprises.
This is not intended to be a complete list and is somewhat generalist in approach. What it is intended to do though, is provide the reader with the opportunity to identify their business (more or less) as being similar to one or two of these personas. So as we explore the operational challenges further, we can then call out what the risks and opportunities are for each of these persona types.
DAMS: Digitally Ambitious Mid-Size
DAMS, in theory, should have the most to gain when moving to headless. They have been trading online for some time, using a now rapidly ageing mid-size ‘shop in a box’ or monolithic solution and know they have to re-platform soon. They need to evolve their brand and product to engage a younger audience in new ways and via new channels but are limited by size or internal technical team and available budget. They find themselves looking enviously towards peers who are moving to headless but struggling to make the business case internally. Therefore, they will probably end up moving to the next-gen ‘shop in a box’ as the technical and operational overheads of headless are likely to be too much.
VIMS: Vertically Integrated Mid-Size
VIMS, have very knowledgeable and able IT teams. Whether they are startups or some years down the road, even though the business is trading successfully online as a pure-play provider of products and services and moving to the next platform iteration; their IT people and UX designers will be curious, self-learning engineers who are embracing automation. They will be genuinely interested in solving the operational problems that come with the move to new technologies.
They are actively engaged in solving the operational challenges and will be eloquently describing this to the business while investing/burning significant time and effort in doing so. Probably for much longer than they should be, as it can hold up delivering the business value that is needed.
LETS: Large Enterprise Tech Savvy
LETS, in theory, have the people and the knowledge capital to get the operational challenge right. They will probably have paid for some external consultancy to help ratify the direction of travel and have set about putting in place what they need from an operational standpoint. They will be progressing nicely but learning along the way that the sheer complexities in their target architecture and the need to manage a significant transition phase, means that everything is taking considerably longer than they thought it would. During this time the goalposts have also been moving as the business needs continue to progress and the decisions made at the start may be proving rapidly outdated as the business responds to external pressures and challenges along the way. LETS will organise themselves using new team structures, aligned to the emergent ‘tribes and squads’ Target Operating Model.
LENTS: Large Enterprise Not Tech Savvy
LENTS have significant opportunity to benefit the business by a move to headless and microservices and should be able to make the budgetary arguments stick. They are probably characterised by deeper silos between business and IT, with fewer skills and expertise available in the team. It may well be the case that it’s the business, not IT, making the case for a move to headless and microservices, as they are seeing the benefits of being reported by peers working for competitors or in other verticals. LENTS are potentially quite vulnerable businesses due to their lack of agility and innovation. They may have significant revenue but they need to modernise to hold on to this as they find themselves increasingly at risk from startup disruptors and other large enterprises that are more LETS; and even from DAMS who are coming after them, benefiting from the fact that they struggle to drive rapid change. LENTS will struggle to handle feature development, never mind the operational changes needed for a move to headless, so will be entirely reliant on third-party service providers.
Key Operational Considerations
If you are about to start your move or are somewhere in your journey, what are the key operational considerations that you should be addressed regardless of whether you favour a serverless or abstracted architectural model?
We have identified eight key areas that will ensure success on the journey to becoming cloud-native:
- Continuous Integration and Deployment
- Seamless Orchestration and Consistent Application Definition
- Service Discovery
- Networking and Security
- Developer Enablement
- Container Registry and Runtime
To manage and maintain a stable and scalable suite of services across a busy and well-used enterprise. Containerisation is often key (where this makes sense over a Serverless approach) to ensuring that the services are easily deployable to a variety of targets and that they can be easily scanned for a variety of vulnerabilities and potential issues.
Docker has established itself as the de facto standard in the containerisation world. The ability to create a minimal instance of a machine dedicated to one function that you specify is extremely powerful.
There is a need to embrace a containerised approach to the headless paradigm to deploy an abstraction layer, which realises a business' need to present their enterprise through a series of well-defined services. We have found in Docker, the ideal level of functionality and maturity to ensure that the highest levels of resilience, scalability and performance can be achieved.
Continuous Integration and Deployment
Rapid and flexible delivery of high-quality software is of massive importance when it comes to an enterprise, with many moving parts and ambition for high flexibility and agility. The ability to deploy code that you are confident and sure of to production, on a daily or even hourly basis, is a target any enterprise should be aiming for. KATA leverages the Jenkins build ecosystems and has a variety of plugins that facilitate a well defined and smooth development and operational workflow. This aids the developer and DevOps engineer, rather than getting in the way.
Generate verification, integration and deployment pipelines that are intelligent enough to take into account secrets management and image creation. Registration is imperative of any functional and robust CI/CD operation.
Orchestration & Application Definition
Defining discrete services that scale under load and interoperate with each other is a difficult challenge and one that can be simplified through leveraging Kubernetes and Helm. As more and more services come online in an enterprise, so does the inherent complexity in that system increase geometrically. With one service, there is only one communication pathway and one ‘thing’ to monitor. As you introduce a second service, you have doubled your workload. A third service increases the communication pathways by three times, and so on.
To ensure that these services are adequately monitored, that they respond gracefully to both load and failure and that they will be amenable to upgrading and debugging, we need to have good means of defining a service and to orchestrate its position in the family of services that make up the cluster. Generate this configuration and you will have all the controls required to define, build, deploy and operate your services at scale.
One of the most fundamental requirements of a service-oriented application that has many moving parts is the ability to monitor the entire enterprise in a homogeneous fashion. This means ensuring that each and every service provides relevant, timely and easy to consume information to those who can react to the situations they describe.
Prometheus and Grafana are cloud-native applications for monitoring, with a variety of custom charts to help analyse every aspect of the running system. Jaeger will provide service traceability for every element of the application deployed to the cluster.
In a service-based ecosystem, cross-cutting concerns such as service discovery, service-to-service and origin-to-service security, observability and resilience tend to be deployed via a shared asset such as an API gateway or ESB. As an enterprise grows in size and complexity, it can become more difficult to understand and manage. This is where a service mesh shows its colours. Linkerd can be employed to configure and control load balancing, traffic control and routing; as well as policies and access controls with rate limiting and quotas, service discovery and linkage to the monitoring and tracing components in the system. Each service and component automatically becomes part of the cluster's wide mesh to take advantage of the manifold benefits Linkerd provides.
Networking & Security
Flexible networking offers a more flexible architecture. The ability for all the different services to accurately find each other and cope with failure means that time spent ‘keeping the lights on’, is delegated to the software and the engineers can concentrate on more business advancement.
To be confident in everything that is being created and not necessarily relies on the ‘magic’ behind the scenes, you will need to be able to prove that what you are expecting to be built is what is actually being built. This requires a set of checks and balances that the Open Policy Agent provides. It ensures all software and hardware is securely created and controlled.
There is a surprising amount of boilerplate code that goes into a homogeneous service-oriented architecture. As a business with constraints on development resources, the ability to move developers between service teams and have them productive immediately is imperative to success. That is achievable by generating the lion’s share of the code and just leaving the feature implementation to the developers. The Open API standard offers a starting point to generating service-based software and being able to abstract some of the more esoteric complexities of the cloud tooling available. This is to ensure that the learning curve for developers is flattened, meaning a quicker time to business benefit is seen.
Container Registry & Runtime
As with networking and security, the ability to be sure that the images being deployed to production are what is expected. A comprehensive cloud-native application must have a secure container registry and runtime. Harbor provides that registry. With a suite of software that will sign and scan the contents in the registry, you can be confident in what is available for deployment to any environment in an enterprise’s estate.
Delivering on the ‘the last re-platform you’ll ever need to make’ promise
The intention here is not to put you off making the move to the new API driven, SaaS world. There are some amazing technologies out there that when used together can have a really positive impact on your business. They are increasingly focused on very specific parts of the customer experience you want to build. Mixing together IoT, conversational or visual search, AI and ML, cross-channel commerce, inventory and order management engines; as well as online to offline content and promotions and payment by facial recognition, this list grows day by day. Many new startups can enter the market using cloud tech to rapidly launch very exciting products that may well be game-changers for any business.
The API driven economy, empowering the move to a service-oriented architecture away from the monolithic suite based offerings of old, brings with it a bright new future. In this new world, businesses will plug and play with any number of these new SaaS offerings. The flexibility to stand up a quick POC, launch a trial service in a particular geography, build a new app or respond rapidly to unpredictable external threats and challenges, all justify making the move.
Your new platform will be so inherently flexible you’ll never need to re-platform again. It is possible to achieve the dream of ‘the last re-platform’. However, learn from the mistakes of others. Being prepared and realistic about the internal limiting factors to achieving this nirvana will stand you in a much better place to getting this right. So spend a moment considering your next move, as it could well make the difference between success and failure.