CI/CD and deployment
Getting changes to production safely and often.
What a pipeline should do
beginnerContinuous integration runs the checks on every change, build, tests, lint, type check, so breakage surfaces in minutes rather than at release. Continuous delivery keeps every passing commit deployable. Continuous deployment goes further and ships it without a human deciding. The three are often used interchangeably and describe different amounts of courage.
The value is proportional to speed, and the threshold is roughly ten minutes. Past that people stop waiting, start batching changes, and the benefit of small deploys evaporates: each release is larger, the failure is harder to attribute, and the rollback takes more with it. Pipeline duration is a product decision rather than an infrastructure detail.
Getting there is mostly ordering and caching. Run the cheap checks first so an obvious failure costs seconds, cache dependencies between runs, run independent jobs in parallel, and only run the expensive end-to-end suite where it earns its place. A pipeline that runs everything in sequence because that was the order it was written in is the common case.
What belongs in the pipeline is anything the project claims to be true. Formatting, types, tests, licence checks, a build that actually produces the artefact you will deploy. Every rule enforced by review and not by a check is a rule that holds until the day someone is in a hurry, and the checks that pay for themselves are usually the ones written after that day.
Flaky tests deserve a policy rather than a habit. A test that fails randomly trains everyone to press retry, which is the same as having no test, and the cost is not the flake but the credibility of every other failure. Quarantine it, fix it or delete it, and treat a rising flake rate as a defect in the pipeline rather than as weather.
The build must be reproducible and the artefact must be the one that ships. Building once and promoting the same artefact through environments is what makes staging meaningful; rebuilding per environment means staging tested something adjacent to what production runs, which is a subtle way to make the whole pipeline decorative.
Why build one artefact and promote it rather than rebuilding per environment?
Evidence · 1 source
- 01DORA: the four key metrics
That lead time, deployment frequency, change failure rate and time to restore are the measured outcomes, and that they correlate with performance better than any process choice.
Answers are generated and can be wrong. The topic above is the reviewed version.
Blue-green, canary and rolling
intermediateRolling replaces instances gradually: take one out, start the new version, wait for it to be healthy, continue. It is simple, needs no extra capacity beyond one instance, and has one requirement that people forget until it bites, which is that both versions run simultaneously and must therefore be compatible with each other and with the schema.
Blue-green runs a complete second environment and switches traffic in one step. Rollback is switching back, which is as fast as a change to a load balancer, and that speed is the whole point. The costs are double the infrastructure for the duration and the fact that the database is usually shared, so the schema still has to work for both sides and the instant rollback does not extend to data.
Canary sends a small share of traffic to the new version and watches before proceeding. It is the strongest default because it fails small: a problem appears on one per cent of requests rather than on all of them. What makes it work is the automated decision, which means metrics good enough to compare the two populations and a rule for aborting that does not need a human awake.
Progressive delivery is the same idea generalised. Route by user, by region, by internal staff first, and increase in steps with an automatic hold when error rates or latency diverge. Cells make this natural, since a cell is already a hard boundary, and the deploy becomes cell by cell with the blast radius decided in advance.
Every strategy needs a rollback that has been tried. A rollback path nobody has exercised is a plan rather than a capability, and the moment to discover that the previous image no longer starts, or that the migration cannot be reversed, is not during an incident. Rehearsing it is cheap and the alternative is expensive exactly once. Finally, decouple deploy from release with flags where you can. When the code path can be turned on without shipping and off without redeploying, the deployment strategy stops being the only lever you have, and the fastest rollback available becomes a configuration change rather than a pipeline run.
What must be true for a rolling deploy to be safe?
Answers are generated and can be wrong. The topic above is the reviewed version.
Database migrations without downtime
advancedSchema changes and code deploys are not atomic, and cannot be. There is always a window in which old code runs against the new schema, or new code against the old one, and every migration strategy is a way of making that window harmless rather than avoiding it.
Expand and contract is the pattern that works. Add the new column, deploy code that writes both old and new, backfill the existing rows, switch reads to the new column, and only then, in a later release, drop the old one. Each step is safe with either version of the code running, which is what makes the sequence longer and the outage shorter.
The backfill needs its own care on a large table. Update everything in one statement and you hold a lock and generate a write burst that replicas struggle to apply; batch it in chunks with a pause between them and it takes longer and disturbs nothing. That patience is usually the difference between a migration nobody noticed and an incident report.
Locks are the second failure mode and they are database-specific. Adding a nullable column is instant on a modern Postgres, adding one with a volatile default is not. Changing a type rewrites the table. Building an index the ordinary way takes a lock that queues every query behind it, which is precisely why the concurrent variant exists. Knowing which operation is cheap on your engine is not optional knowledge for anyone deploying to it.
Lock queues are worse than the lock itself. A migration waiting for a long-running query holds its place in the queue, and everything arriving afterwards waits behind it, so a table that was merely busy becomes unavailable. Setting a short lock timeout means the migration fails quickly instead of taking the table with it, and failing a migration is a much better outcome than an outage.
The rule that follows: every migration should be reversible, tested against production-sized data, and separated from the code that depends on it. Splitting a rename into three deploys feels slow, and it is the only version that stays up.
Why set a short lock timeout on a migration?
Evidence · 1 source
- 01PostgreSQL: ALTER TABLE notes
That adding a column with a constant default avoids a table rewrite, while a volatile default does not, and which lock level each operation takes.
Answers are generated and can be wrong. The topic above is the reviewed version.
Feature flags
intermediateA feature flag separates deploying code from releasing behaviour. Code ships dark, is enabled for a cohort, then for everyone, and turning it off is a configuration change rather than a redeploy. During an incident that difference is minutes against a pipeline run, which is why flags are how teams deploy on a Friday without pretending it is a normal Tuesday. There are several kinds and conflating them causes most flag problems. A release flag is temporary and exists to be deleted. An experiment flag exists to be measured and then decided. An operational flag, a kill switch for an expensive feature, is permanent by design. A permission flag is really entitlement and probably belongs in the authorisation model rather than in a flag system.
Flags accumulate, and each one is a branch that doubles the number of possible behaviours. Ten live flags is a thousand combinations, of which you test perhaps three. That is why deletion has to be part of the process, with an owner and an expiry date, rather than something everyone intends to get to once the launch settles.
The flag service becomes a dependency on the request path, so it needs the treatment of one. Cache the values locally, define what happens when the service is unreachable, and make the default the safe path rather than whatever the client library returns for an unknown key. A flag system that takes the product down when it is unavailable has inverted its own purpose.
Flags are also a testing problem. The combination you ship is the combination someone experiences, so at minimum test the on and off state of anything currently in flight, and be honest that beyond a handful of concurrent flags nobody is testing the matrix. That is another argument for deleting them promptly rather than a reason to avoid them.
Used well, they change the shape of a release: small, frequent, dark deploys, then a separate decision about exposure, with a rollback that does not need the pipeline. Used badly they are a permanent second configuration language that nobody owns and nobody can safely remove.
Which kind of flag is legitimately permanent?
Answers are generated and can be wrong. The topic above is the reviewed version.