“Should we use AI for this?” is the wrong question. The question you should ask yourself is: “Can AI help me with this?”
AI is a tool. A versatile tool. This tool should make me better at my work. How do you know if a tool makes you better?
In this article we propose a basic framework to assess if using AI is a good idea. For this, we use five metrics: Quality, effort, knowledge, feedback loop and risk. The goal is not to maximize AI usage, but to make its users better at their job.
The Metrics
- Quality
If you use AI for ‘x’, will the quality of your product/process go up, down, or stay the same?
This includes correctness, maintainability, clarity, documentation, and much more.
- Effort
If you use AI for ‘x’, will human effort go up, down, or stay the same?
This is all the work that a person needs to do. Including prompting, correcting, and implementing the results of the AI into your product/process.
- Knowledge
If you use AI for ‘x’, in the long term, will the knowledge of individual team members and the team as a whole go up, down, or stay the same?
This is the most important metric. It depends on how you position AI in your process. In the before-times, you became a good engineer by doing your work and learning along the way. AI can aid this learning process or break it.
The same thing applies to teams. Teams don’t fail because they’re slow. They fail because they don’t understand their own .
- Feedback Loop
Will using AI for ‘x’ create a positive or negative feedback loop?
In general, more knowledge leads to better decisions, which leads to higher quality, which reinforces knowledge. Or the opposite can happen: Less knowledge leads to worse decisions, which leads to worse quality, which degrades knowledge further.
- Risk
If you use AI for ‘x’, what is the risk on production: low, medium, or high?
Metrics like these can feel very abstract until you start applying them. So let’s look at some example situations where AI is used in different ways.
Example 1: A Junior Developer Using AI to Build Features
Scenario: A junior developer uses AI to generate production code for new features.
Quality: Generally trending down. The code usually runs, but subtle bugs, missing edge cases, and poor design choices slip through. Juniors often lack the experience to reliably spot these issues.
Effort: Less effort in the short term. More effort long term since poorly understood code is harder to maintain.
Knowledge: Down. The junior learns how to prompt, but not how to be an engineer. By outsourcing their work to the AI, they skip the learning opportunities they would otherwise use to grow.
Feedback loop: Negative. Less knowledge leads to more AI reliance, which leads to even less understanding. Over time, AI reliance increases, code quality drops, and learning stops.
Risk: AI written code ships to production resulting in more (recurring) incidents at all risk levels.
Verdict: A short-term productivity gain that quietly creates long-term problems in quality, knowledge and personal growth.
Example 2: A Junior Developer Using AI as a Learning Tool
Scenario: A junior developer works together with AI to understand code, explore approaches, and get feedback. A senior developer is available for further guidance and coaching.
Quality: Up over time. The junior builds knowledge by engaging with AI responses rather than blindly applying them. Simple mistakes get caught earlier and coding skills grow.
Effort: Down for both the junior and the senior. AI takes care of routine teaching like explaining code and reviewing drafts. This frees seniors for meaningful coaching and teaching higher-level concepts.
Knowledge: Up. The junior learns to code better, review results, and learns faster.
Feedback loop: Positive. Growing understanding leads to better implementations, which leads to a better product.
Risk: Low to medium, and decreasing over time. AI review catches mistakes early and the senior catches the rest.
Verdict: AI used this way amplifies growth instead of outsourcing it. The junior develops faster, the senior can focus on meaningful guidance, and the team builds sustainable skill rather than a dependency on AI.
Example 3: A Senior Developer Using AI for learning
Scenario: A senior developer uses AI to explore design options, draft code changes, and generate scaffolding.
Quality: Up. Seniors have the skill to reject bad ideas and refine good ones. AI expands the solution space without replacing judgment.
Effort: Down. Scaffolding, comparisons, and first drafts are faster. The hard thinking still happens, just with less friction.
Knowledge: Up or neutral. Strong pre-existing mental models mean AI output is evaluated but not trusted blindly. This can reinforce understanding.
Feedback Loop: Positive. Better designs lead to cleaner systems, which are easier to reason about, which makes both humans and future AI usage more effective.
Risk: Low to medium. Seniors work on high-impact areas, but also add guardrails: tests, reviews, and incremental change.
Verdict: This is a good use of AI. It amplifies expertise instead of masking its absence.
Conclusion
Depending on how you use AI it can produce different outcomes.
When AI is deployed only to speed up, you’ll often miss out on knowledge and understanding of what your product does. This means that you’ll become more and more reliant on AI and you’re (eventually) not able to correct it when it inevitably makes a crucial mistake.
Instead, AI should be used to amplify your abilities, extend your knowledge base, and increase human understanding of what we’re developing. In that way you’ll get the best of both worlds.
Many people talk about “human in the loop” as a way to keep AI in check. This is not enough. A disengaged human who rubber-stamps AI output is just slow automation. What you want is a brain in the loop: someone actively reasoning, questioning, and building knowledge and understanding.
Do you have questions about using AI in your testing work? Feel free to get in touch with us.
