Release Line

The Main Line contains the code from all completed Task Branches. When you release software, you want to know what you released for historical and support reasons. This pattern describes a structure that can help you identify what was released and enable emergency changes to currently released code in the rare instance that you cannot deploy from the Main Line quickly enough to meet business needs while deploying code from the Main Line_

Identification and Handling the Unexpected

Releasing software should be low overhead: when work is merged, you release software from the HEAD of the Main Line. As you deliver code, you want to know what code was in each release to allow you to reproduce the identical configuration when you want to test or debug software. Knowing the history of a code line and identifying what code is in a release are essential functions of a software configuration management process and also support any audit processes you need to follow.

Releasing from the Main Line is possible because we have steps like Unit Tests that give us some confidence that each module works as we develop. We also run Integration Tests as changes merge to the _Main Line_to give us some added confidence. Even with a good testing discipline, errors are still possible. And not every team’s test practices are where they’d like them to be.

Having frequent releases supports Main Line development as well. The frequency can range from:

• After each merge to the Main Line. This requires that you have a high confidence in your test and release process — and a highly automated one.
• Periodically, for example, daily or every other day, regardless of your development iteration length.
• At the end of a development iteration.

Regardless of the frequency, you might find value in doing some pre-release testing of the release-candidate code before deploying to production for a brief period of time.

Should you discover an error in the release candidate, you can:

• Fix the error on the Main Line and release when the issue is fixed
• Defer the fix until the next release interval
• Apply a change to a release candidate branch

The branching model might seem more straightforward, but it:

• Results in a divergent workstream since now are adding the release candidate branch to the Main Line or Task Branches
• Increases the time for other work to be released because you are focusing on the features in the current release candidate

The challenge is to balance the desire for a single development stream with rapid, low-error deployments.

Once you deploy, even with the best testing, an error might be observed in production. When this happens. Some options are:

• Identify how to mitigate the problem and wait until your next periodic release to address the issue. Given a frequent (weekly) release schedule this might be suitable for some errors
• identify and fix the issue based on the latest Main Line and deploy the resulting code.

While releasing any merged code at any time is a desirable goal, your team might not be able to do this because business or other constraints limit the rate at which you can deploy code from the Main Line to production.

• Changes in the Main Line might have made the fix more difficult to deploy quickly; for example, an extensive refactor was in progress, and a small fix is now at higher risk.
• Your current process is that you release at the end of a development iteration, and you are not yet confident in your ability to support or thoroughly test work in progress before the end of the cycle, allowing for immediate deployment of the current merged code plus any fix
• Your software follows a regulated process that requires that differentiates between fixes and added features.
• There is some third-party approval review required for release, and you don’t want to block merges to the Mai Line while this is happening, for example, an App Store Approval Process.

None of these situations is ideal. Not all errors require rapid emergency fixes. However, some changes cannot wait for the next release at the end of the planning cycle.

You want a low-overhead solution that balances:

• Speed: the ability to provide a rapid fix for code that is in production
• Flow: having as few development streams as possible so that you can focus on delivering value, and refraining from adding high overhead processes.
• Stability: discovering errors in production affects users and the team

Branch on Release

** When merging deploying code to any environment, create a tag identify that you can use to create a Release Line. Create a Release Line branch before deploying. You can add changes to the branch should you need to deploy a fix before the next regular release cycle, but avoid applying fixes to release candidates. Limit the number of active release lines to 1. Patch releases should be rare.**

A Release Line starts from the Mainline. Its purpose is to identify a released version of the software. Usually, A Release Line should be a marker: a code line that identifies what was released or deployed with no commits added and which ends its active life with the next release. The Release Line could be deleted at this point, but we often keep it around for review, audit, or compliance reasons. Limit the number of deployed release lines, ideally, to 1. A Release Line is a unique branch type because it’s a branch you rarely want to add commits to. A Release Line also has a very restrictive Codeline Policy

For validated and other systems that require deployment gates, a Release Line is a mechanism to balance an agile continuous integration workflow with controlled software release. After each commit to the Main Line, code is deployed to a pre-production environment using the usual deployment workflow. In this case, minimize the number of active Release Lines.

After a merge to the Main Line passes any required checks:

• Tag the version with a meaningful name indicating a version, and create a branch based on this tag, the Release Line
• Build an artifact with the code from this tag.
• Make the artifact available for whatever final review is necessary, such as Final User Testing or an App Store Review.

This process can occur after each merge or can happen at the end of an iteration. Ideally, you have a process that deploys code to a pre-release environment with every merge so that you can exercise your release process and always have a release candidate available,

Once the final reviews have passed, you can release the code to end users. For example, deploying a web application, releasing it in an App Store, etc.

If you discover a problem in the release candidate code, you have some choices:

• Fix the error on the Main Line and repeat the normal Release Line process. Since the release started from the head of the Main Line, and your verification should have been completed quickly (within a day or so), this should usually be as quick, or quicker, than alternatives and avoids the creation of a new parallel work stream. This is the best option.
• Prioritize the issue for the next release. This is a good choice if the problem is low risk and you have a frequent release cadence.
• Fix the release line and integrate the same change into the Main Line. There are rare cases when this is the best alternative, and since it created a new work stream, splitting the teams’ attention, you should avoid this approach.

If you discover an error after the code is released, the options are similar to those for fixing release candidate errors, but perhaps with some added business constraints:

• Fix the Error on the Main Line and prioritize a new release. This works if you can keep your code in a shippable state, and can deliver code mid-development iteration. Not all teams can do this reliably.
• See if you can mitigate the issue and defer the issue to the next release, adding the work as a priority item in the backlog for the current or a future iteration.
• Provide a fix as an update to the Release Line. This might be your best option when it provides the fastest path to delivering a critical fix.

Mitigation can be best, as it avoids creating a new work stream, and injecting a fix onto the Main Line is also a good option if it works for you; there are cases when a patch release off of the Release Line is the best option. For example:

• For a business or life-critical error, fixing the branch is significantly faster — including work including any regulatory or other reviews — than releasing from the Main Line.
• Your software must follow a periodic release process for regulatory or contractual reasons, making adding code changes infeasible and using Feature Flags or a similar approach is not an alternative.

Whenever you find yourself doing a patch release, take time afterward to consider if there are process changes you can make to have the usual Main Line approach be a better option.

In the rare event that you need to make a fix to a Release Line :

• Create a branch from the release tag if one does not exist.
• Checkout a “fix” branch
• Make changes and mere to the release line after a review/test cycle
• Incorporate the changes to the Main Line either:
  • Merge the post-deployment version of the Release Line to the Main Line
  • Apply analogous changes to the Main Line using a new Task Branch. There may be some cases where the changes are no longer relevant to the code on the Main Line.

Pre-Release Environments

A common practice for teams that don’t do continuous deployment to production is to deploy to a pre-production environment for internal use and testing. Each merge to the Main Line could be deployed to a “developer” environment, for example, while periodic releases are deployed to a “staging” environment for user acceptance testing. This is a valuable way to exercise the deployment process as long as the intervals between “staging” releases are kept short.

If you use pre-release environments, each environment branch is a release line. If you have more than one (dev, stage, prod) ideally each should start from the same point on the Main Line. If you are using a time-boxed development cycle, aim to deploy to the production environment one or more times per cycle to avoid pre-release stages from diverging from each other.

Cautions

A Release Line isn’t intended to make the final leg of your agile development process Waterfall. While using the Release Line for identification is valuable, and having a safety net for patch releases is useful, you don’t want to make patch releases a habit. Be thoughtful when releasing a patch (or hotfix), and aim to reduce the interval between deployments as much as your business environment allows.

Each patch release should motivate you to make them less necessary. Ideally, see if you can identify mitigations or other workarounds that allow you to delay the fix until the Main can be deployed. Even in regulated environments, you can use a process that allows for rapid deployments from the Main Line.

Do not use a Release Line to “stabilize” code (in the manner or a Release Prep Codeline in SCM Patterns. Unless there are exceptional circumstances, any errors you find in pre-deployment testing should mean moving on to the next release candidate and improving your upstream testing.

It’s worth repeating that:

• Release Lines should rarely accept changes
• There should be only one release line, to the extent possible

Next Steps

Document (and automate) the Codeline Policy that applies to the Release Line