The front-end for "Employee Referrals" — the department where I work — at Radancy is a single page web application, served as static assets by nginx, which also acts as a reverse proxy.
The problem
Historically, the front-end team at "Employee Referrals" has followed a practice
of working on features in long-running git branches, iteratively adding small
changes to the branch until the feature is complete, at which point they merge
it. While this (somewhat) worked when these front-end developers were part of a
dedicated and siloed team, it doesn't really work well with our new team
structure — cross functional teams where front-end and back-end developers work
together on product features. The need to wait for merging these long-running
branches to the main branch before seeing the corresponding features on our
pre-prod environment qa resulted in significant delays. This meant that it
took quite a long time before other team members — product owners, testers,
other devs — could test out new features and provide feedback. I wanted to get
rid of this delay by allowing previews for front-end feature branches.
These feature previews needed to be easy to use with very low friction for everybody involved. This is the workflow I ended up implementing.
The solution
Each team that contributes to the front-end codebase gets a few "deployment
slots", named in a certain way, eg. feature-w-1, feature-w-2, feature-r-1,
feature-r-2, and so on. The w and r in the slot name stands for the team
name. A developer in these teams needs to prefix their git branch with the name
of the feature slot, eg. feature-w-1/make-scrolling-ultra-smooth. Once this
branch is somewhat ready to be previewed by other team members, the developer
needs to create a pull request with this branch, which will then trigger a
deployment of the codebase in the branch.
Team members who need to preview this feature need to have a browser extension installed, one that injects a custom header to all requests made to a certain domain. They can choose a deployment slot by simply changing the value of this header to the name of the deployment slot.
How it works behind the scenes
The normal setup
In the production setup — one without this feature preview functionality — nginx
runs as a fargate task on AWS ECS, getting traffic from an Application Load
Balancer, which itself acts as an "Origin" for a Cloudfront distribution. The
CICD pipeline (on Jenkins) for this repository builds docker images and pushes
them to dockerhub, which are then used for the ECS task definitions. Fargate
service updates are handled by cloudformation — Jenkins just runs an aws
cloudformation update-stack command at the end of the pipeline.
The web application can also be served on a custom domain (owned by a customer)
by having it be served by a dedicated Cloudfront distribution that has a CNAME
or ALIAS DNS record pointing to it. However, this means that feature preview
deployments need to work with custom domains as well.
Infrastructure for feature previews
For the feature preview deployment setup, an ECS fargate service is created for each deployment slot, which is supposed to run an ECS task with a container running the codebase for the git branch corresponding to that slot.
Routing rules are added to the Application Load Balancer on qa, which route
based on the value of the header X-1brd-Feature to the target group associated
with that deployment slot. The desired count of the service for a deployment
slot can be set to zero if it's not needed.
As mentioned above, our web application — irrespective on whether it's on a
custom domain or our organisation domain — is served by Cloudfront. This HTTP
header based approach works with the existing Cloudfront setup as it can simply
forward the X-1brd-Feature header to the ALB. The only extra configuration
needed on the Cloudfront distributions is to include the header in the cache
policies being used.
Continuous delivery for feature previews
Jenkins runs a separate pipeline for pull requests where the change branch matches the regex for the feature names.
// Jenkinsfile
when {
expression {
return env.CHANGE_BRANCH =~ /^feature-[wr]-\d{1,2}\//
}
}The docker image built in this pipeline is tagged with a string comprised of the branch name, and the build number.
// Jenkinsfile
environment {
CHANGE_BRANCH_DOCKER_SAFE = env.CHANGE_BRANCH.replaceAll('/', '-')
RELEASE_VERSION = sh(script: "echo \"$CHANGE_BRANCH_DOCKER_SAFE-$BUILD_NUMBER\"", returnStdout: true).trim()
}This pipeline then updates the stack that manages the feature deployments ECS cluster, only updating the service that corresponds to the feature deployment in consideration.
Future
The primary idea behind adding feature previews is to increase visibility in the features our teams are building, and enabling feedback loops so bugs/discrepancies are caught early in the software development lifecycle process. As this workflow is new in the whole "Employee Referrals" department, we'll have to proactively make use of preview deployments to get the most out of them.