CI & CD

In part 1 of this series, we explored two strategies for delivering software: One we called zero-gap: closing the gap between submission and testing early, at the cost of latency. One we called “letting the gap trail”: chasing failures and reverting them after submitting changes, at the cost of correctness. We explored how the latter works at scale for huge companies like Google, but we did not explore how to scale when latency becomes too much of an issue with the former, zero-gap strategy. If your CI takes 30 minutes to run, the whole company can only merge roughly 16 PRs in an 8 hours working day. ...

In this Test Automation series, we investigate ways to optimise code integration and delivery speed without trading off software quality. In a previous article we explored the “when” to run tests during continuous integration. An equally important question to answer is “what” tests to run. What are some industry guidelines we could apply to a codebase with thousands or millions of tests? One line, thousands of tests In a test setup that grew organically, without any optimizations, a developer edits one component file. The CI queues the whole test suite (let’s imagine 10,000 tests). ...

The speed vs quality dilemma Last week I laid out one of the CI/CD (Continuous Integration & Continuous Delivery) engineer dilemmas: improve quality with more test coverage, or improve the speed of delivery. Brute-forcing both by throwing more machines at the problem works for a while, then stops working when the test suite runs for hours and rarely comes back green on the first try. But speed and quality don’t have to trade off against each other, and there are ways to speed up integration and delivery, while aiming for high quality. ...

The Test Automation Engineer dilemma Your company is growing (yay!). You have more customers. You add more features and support more use cases. You can’t afford breakages and downtimes anymore, so you decide to improve the robustness and quality of your testing. In parallel, supporting your growth means hiring more developers, who need to ship fast, with early feedback and limited context-switch, while also sharing a pool of finite test and deploy resources. Integrating and deploying faster (latency) vs increasing test coverage (correctness) is often the dilemma Test Automation Engineers face in such cases. ...