Knowing What You're Shipping

We use Jenkins for continuous integration and continuous delivery of software in my team at Radancy. Jenkins then sends a notification to a Slack channel (a dedicated channel for monitoring needs) when a deployment is through. When I joined Radancy last year, these notifications looked like this:

As is clear in the image above, these notifications do not convey a lot of information about the change that is being deployed. The most useful piece of information in these notifications is a link to the Jenkins run. We use Bitbucket for source code management — we’re migrating to Github soon — which doesn’t have the best user experience for viewing commits, and doesn’t have a feature like Github Actions, so I was stuck with using Jenkins to improve these CICD notifications.

The complete Jenkins pipeline for a commit on the main branch looks like this:

As is clear from the image above, we deploy code to 3 deployment environments: qa, staging, and prod, with a manual action needed to go from qa to staging, and from staging to prod. Behind the scenes, Jenkins runs a aws cloudformation update-stack command in the “deploy” stage.

This manual action sometimes leads to these three environments getting out of sync — there may be a blocker that prevents the change from being deployed ahead, or, maybe somebody just forgot to push their changes ahead. If a significant chunk of commits pile up between qa and the last commit to prod, the person who has to push the latest commit to prod now has to be aware of any maintenance/upkeep needed for the other commits. On top of that, since changes to our cloudformation stacks need to be applied manually — either via the cloudformation dashboard, or the aws cli — this pile up of commits can cause bugs/issues when one (or more) of them contains a cloudformation change, and the person deploying the latest commit forgets to apply the change.

So, keeping the above in mind, I needed our CICD notifications to convey the following details:

For PR merges and the subsequent deployment to qa:

1. Who’s commit is being deployed?
2. What files got changed?

For deployments to staging and prod:

1. The changeset of commits being deployed
2. What files got changed in this changeset?
3. Did infrastructure change since the last commit to this environment?

The primary idea behind this is that having more information in these notifications promotes more visibility into what — and whose — code changes are in the queue to be shipped to production, enabling the engineers to take fast action if something breaks.

In general, I’m concerned with the following 5 “stages” in the entire CICD pipeline:

1. PR merge
2. Deployment to qa
3. Changeset for staging
4. Deployment to staging
5. Changeset for prod
6. Deployment to prod

PR merge

These notifications need to convey that a PR has been merged, and is getting lined up to be deployed to qa. It should also let the team members know not to merge another PR till this one has been successfully deployed. They also tag the relevant person so they get notified on Slack.

Deployment to qa

This notification is for team members to know that qa is now free to receive new changes, and for the concerned person to test out their changes.

Changeset for staging

This is where we start seeing a major benefit of these notifications — they clearly indicate which commits have piled up since the last deployment to staging. Team members can also indicate that their changes are ready to be deployed to staging by simply marking this message with a ✅ sign.

Deployment to staging

These notifications are similar to the ones for qa, with the difference that they convey updates about a changeset of commits, rather than a single one.

Changeset for prod

These notifications are the most important ones as they indicate the difference between the last deployment to prod and the changeset of commits that have piled up since then.

Deployment to prod

Finally, these notifications provide details about the changeset being deployed to prod. If the code changes were well tested on qa and staging, this should be pretty straightforward. Regardless, having more insight into what we’re shipping out to users is always helpful.

Next steps

I want the Jenkins pipeline to interrupt deployment to staging or prod if the actual cloudformation state is different to what the commit being deployed says it should be (via the stack template file). Once this difference has been detected, it should notify the Slack channel so an engineer can take a look at the differences, and take appropriate action. I’ll maybe add an override to this interruption, for the rare cases where we’d want the deployment to go through despite a difference in the real stack and the stack file.