You might be using AI coding agents already to write your automated tests. In my previous blog, I showed how coding agents combined with a skills setup can help create E2E tests reliably, while following the conventions and style of your project. But what if we could use these agents not just to write tests, but to actually test our applications?It allows us to explore new functionality, investigate edge cases, and get feedback while we’re still developing a feature.
In this post we’ll explore agentic testing and why it should be part of your testing strategy. I’ll share what I’ve learned so far, and where I think these agentic quality checks can add value. Hopefully, it will inspire you to start experimenting with them in your own project as well. 😃
It’s time to start adding autonomous quality checks in your project next to the scripted automation.
What is agentic testing?
Scripted test automation is very good at following a path we already know.
Action > Action > Assert
Agentic testing agents don’t need that path. They can inspect what is happening in the application and decide what to do next. They run in a loop, taking actions and evaluating every time until they reach their goal.
Action > Make LLM call, evaluate current state towards goal > Next Action
This makes agentic testing very flexible but also it will never be as fast, reproducible and accurate as predefined scripts.
Why agentic testing is worth exploring
Last year I hosted over 20 workshops on agentic AI. I’ve noticed many people want to get started but don’t know where to begin. Some teams are already using AI coding agents to write their automated tests, but very few actually use agentic AI for testing.
I think one reason is that it’s not immediately obvious where agentic testing adds value.
Autonomous agents are particularly useful for testing new functionality while it is still being developed. They can explore a feature, react to what they see, create the required test data, and investigate scenarios that were never explicitly scripted beforehand.
Even though an agent is slower than a scripted Playwright test, it can still provide feedback much faster than waiting for a manual check.
When to run agentic testing checks
I believe this type of agentic testing is especially useful during development itself.
Shift-left testing is all about getting feedback as early as possible. With AI agents, we can now make end-to-end exploratory testing much easier during development.
Instead of waiting until a feature is finished, you can use an agent to validate the implementation throughout development.
I really think this is great for the overall quality. It’s becoming much easier to do this many checks with very little effort before a pull request is even created!
Scripted automation complements this by providing fast, reliable, and repeatable regression checks in CI.
I also like that testing becomes conversational.
While developing a user story, I might ask an agent to test the functionality implemented so far. It will usually start with the main happy paths and report what it found. After reviewing the results, I can ask it to explore additional edge cases, focus on a particular scenario, or investigate something that looked suspicious.
That feedback loop makes exploratory testing much cheaper.
In the past, limited time often meant choosing which scenarios to test and which ones to skip. With autonomous agents, it becomes realistic to explore many more cases.
And this is not limited to functional testing. You can also ask the agent to inspect things such as keyboard navigation, accessibility behaviour, validation states, error handling, or other quality characteristics that might otherwise receive less attention.
I believe a good modern testing strategy should include both agentic and scripted test automation.
This does not require developers to own all testing. In a more traditional workflow, testers can use the same approach once a feature reaches a test environment. Let agents handle execution and exploration while the tester focuses on directing and judging the evidence.
Trusting your AI agent
But AI agents make mistakes, they hallucinate, fill in the blanks and don’t properly follow instructions.
How can we ever trust an AI testing agent?
After running many agentic tests, I’ve learned that providing good context makes agents much more effective. Clear requirements and constraints help them understand what to test and reduce the number of assumptions they need to make.
But that does not mean I automatically trust the result.
Trust comes from evidence.
I want to see which scenarios the agent executed, which paths it followed, what it observed, and why it concluded that the behaviour was correct.
That evidence allows me to decide for myself how much I trust the result.
It also makes it much harder for the agent to take shortcuts. Instead of simply claiming that a scenario passed, it needs to show what it actually did.
This shifts my role from executing every test myself to reviewing whether the checks were sufficient and whether the evidence supports the conclusion.
If the checks are not sufficient, I will ask the agent to look at a few more edge cases, test a specific case, or provide more context to do another test.
My agentic testing skill
What does this look like in a real project?
I have a test-user-story skill that takes a user story or feature description as input and performs the exploratory testing for me. It can also reuse information from our existing scripted E2E setup, such as test data, users, and environment URLs.
The workflow looks roughly like this:
- It receives a user story or description of the functionality.
- It derives a set of relevant test cases.
- It asks which cases should be executed, against which environment, and what type of evidence should be captured.
- It executes the selected scenarios using playwright-cli
- It generates a consistent test report containing the results and supporting evidence.
The template itself is just HTML, together with a small script that fills it with the details from the test run. Whenever a report needs to be created, the agent calls that script with the collected results.
Instead of receiving a different style of output every time, I get a predictable report that is easy to scan and review. More importantly, the report leaves behind an artifact showing what the agent actually tested rather than only its final conclusion.
See the full skill at my GitHub https://github.com/tlolkema/demo-webshop/tree/main/.agents/skills/test-user-story
Conclusion
For me, the biggest lessons with agentic testing are:
- Agentic testing will not replace scripted automation, but it can complement it really well. Scripted tests remain ideal for fast and reliable regression checks in CI.
- Agentic testing adds value by exploring new functionality, testing edge cases, and providing feedback much earlier in development. A modern testing strategy should use both.
- The key is not to blindly trust the agent, but to review the evidence it provides.
- As (quality) engineers, we should embrace agentic testing and the opportunities it brings. Software development is evolving rapidly, and with it come exciting new possibilities.
Let’s embrace these changes and help shape the future of testing!
