218: Balancing test coverage with test costs - Nicole Tietz-Sokolskaya

Python Test

Nicole is a software engineer and writer, and recently wrote about the trade-offs we make when deciding which tests to write and how much testing is enough.

We talk about:

• Balancing schedule vs testing
• How much testing is the right about of testing
• Should code coverage be measured and tracked
• Good refactoring can reduce code coverage
• Is it worth testing error conditions?
• Are rare error codes ok to just monitor?
• API drift and autospec
• Mitigating risk
• Deciding what to test and what not to test
• Focus testing on key money-making features
• If there's a bug in this part of the code, how much business impact is there?
• Performance testing needs to approximately match real world workloads
• Cost of a service breaking vs the cost of creating, maintaining, and running tests
• Keeping test suites quick to minimize getting distracted

Links:

• Too much of a good thing: the trade-off we make with tests
• Load testing is hard, and the tools are... not great. But why?
• Yet Another Rust Resource (YARR!)
• Goodhart's law - "When a measure becomes a target, it ceases to be a good measure"

Sponsored by Mailtrap.io

• An Email Delivery Platform that developers love.
• An email-sending solution with industry-best analytics, SMTP, and email API, SDKs for major programming languages, and 24/7 human support.
• Try for Free at MAILTRAP.IO

Sponsored by PyCharm Pro

• Use code PYTEST for 20% off PyCharm Professional at jetbrains.com/pycharm
• Now with Full Line Code Completion
• See how easy it is to run pytest from PyCharm at pythontest.com/pycharm

The Complete pytest Course

• For the fastest way to learn pytest, go to courses.pythontest.com
• Whether your new to testing or pytest, or just want to maximize your efficiency and effectiveness when testing.