DevOps is a relatively new phenomenon consisting mainly of development and operations practices and an ideology that aims to break down barriers between development and operations. Whether devops is really the solution is another matter.

Here is a sample of some of the key practices often talked about:

  • Continuous integration
  • Configuration management
  • Continuous deployment
  • Infrastructure automation
  • Containerisation

With testing, the phrase often used is “shifting left”. And people always talk about test automation and generally automating as much as possible.

But hang on a minute… is there a real danger that devops is moving us away from good testing? Encouraging us to automate things too quickly (checking before we’ve properly tested)? Stopping us from having important conversations about quality?

Ok, so people would argue that left shifting means we do all the same things but earlier in the process. And that the repetitive things are automated. But let’s look at what might be described as good testing in a devops world.

  • Lots of automated unit testing
  • Lots of automated integration testing
  • Lots of static testing / 3 amigos conversations during backlog refinement
  • Automated tests built into the pipeline
  • Exploratory testing rather than scripted
  • Developers involved in lots of the above
  • Any manual testing done locally before CI gets its hands on things

This might not be exactly what is going on out there, but is probably what people strive for and what a good number of teams do.

But here’s what we are doing less and less of as a result:

  • Actually using the product in anger for sustained periods of time, that lets us really understand the product and expose problems
  • Exercising the product in a range of ways at different times, like real users do
  • Thinking about the product (rather than thinking about how to speed things up)
  • Talking about the quality attributes of the product (rather than the quality attributes of the process)

I get that improving processes can be useful sometimes and that getting things to users faster might be a good thing for the users, but i sometimes think we’ve convinced ourselves and our users that this is a good thing. Really, doing 10 builds a day is actually needed?

My summary. The faster we go and the more we think about process, the less we do good testing.

Advertisements