The right mix between too much and too little software testing

Software testing finds problems before they show up in production, and proves that functions do what they are supposed to. What matters is not the number of tests but their selection: a test run that covers nothing business-critical costs time and still provides no certainty.

Four types of test and what they are good for

The common methods fall into four classes. Each has a different purpose; they do not replace one another.

Unit and module tests

They check the functions of a single component in detail. The test routines are usually written while the module is being developed, before it is integrated into the overall system or goes live. In agile development they are a fixed part of implementation: the quality of a module is thus established before it is rolled out.

Integration tests

They check whether the components of a system work together cleanly, above all the interfaces. Does data exchange with connected systems run without errors? This question becomes more important the more systems are attached to a platform. An integration test therefore usually consists of a whole series of individual test sequences. They also show early on how well a new solution fits into the existing landscape at all.

Smoke tests

They check whether functions and services are present and reachable at all. Two points in time make sense: before the go-live of a new application, to make a final check of the availability of the overall platform, and during ongoing operations. If a component on the server or an external script fails unnoticed, a function is gone without the page looking broken. For the automated runs we use Ghost Inspector, which is also part of our technology stack in other respects.

Manual testing

The most laborious variant, and the only one that shows how an application can actually be used. This is where workflows stand out that are technically correct but cumbersome or illogical. Manual testing is therefore mainly a tool for UX and UI and less for proving functionality.

The right mix depends on the project

How much testing is enough? In our view, that depends heavily on the project. A website project usually has far fewer functions than a web application that maps entire business processes, and accordingly needs less test automation and less development effort for tests.

The guardrail stays the same: all business-critical workflows must be covered. The shopping basket in an online shop has to work, otherwise revenue is lost directly. For rules like this we write unit or Behat tests.

Behat tests describe a scenario in plain language before a line of test code is written, following the pattern ‘Given’, ‘When’, ‘Then’. This means that even people without a development background can read what a test actually covers.

Because the business-critical workflows differ from project to project, finished tests can rarely be reused one-to-one. Rewriting them, on the other hand, is quick, which keeps the effort per project within limits.

What testing actually brings

Putting a new platform into operation without tests is risky in networked IT environments: if a single component does not work as expected, it drags a series of other processes down with it. The benefit is accordingly spread across several levels:

• Quality assurance. A targeted test strategy covers a wide range of problems before they reach production.
• Functional checks. Automated runs check at high frequency whether the configured workflows are available.
• Better usability. Manual tests uncover weaknesses in UI and UX that no automated test finds.
• Cost efficiency. Once set up, an automated test run causes hardly any effort and can run as often as needed.

How we involve you

The goal is not to test as much as possible, but to choose the right tests for the desired result. We work out which ones those are with you: for specialist applications, nobody knows the requirements better than you do.

For this we provide you with your own test environment in which you can check things yourself. The development phase ends with the test and acceptance report.

The mix we use in our own platforms is described in the follow-up post.