Is AI a curse or a blessing for exploratory testing?

AI (artificial intelligence) is hot. This is also true within the field of software testing. Will the developments stop here, or have we only just begun? How can AI help us with exploratory testing? Are there any limitations? And, what questions should you ask yourself before deploying AI in software testing? In this blog, I briefly share my thoughts on the question: Does AI make your life as a tester easier or harder with regard to exploratory testing?

What exactly is Exploratory Testing (ET) again?

Exploratory Testing (ET) is one of the three pillars of our Quality Infected Teams philosophy, namely Exploration. With ET, the creativity, genuine critical thinking, and curiosity of the tester are central. The tester combines these skills to learn as much as possible about the software they are testing. In ET, you are very consciously focused on a part of the product and decide in the moment what the most meaningful path is to discover more about the area you are focusing on. Like an experienced explorer venturing into unknown territory, an exploratory tester navigates the software landscape. They look for clues from various sources, such as the user interface layout, the wording, logs, and infrastructure. Because you do not define all expectations about the product’s operation in advance through procedures or test scripts, you can continuously adjust your strategy based on what you learn as you go. This way, you discover things you hadn’t considered beforehand.

While traditional testing may rely on predefined test scripts and expectations set during development, understanding the context in which the product functions is very important in ET. Can AI help you as a tester when applying ET?

Generative AI Tools

The best-known AI tools are so-called Generative AI tools. Generative AI is a form of artificial intelligence that can produce text, images, and varied content based on the data with which the tool is fed/trained. Training requires one or more datasets as input. Well-known examples of Generative AI tools include ChatGPT, Bard, Scribe, and Dall-E2.

“As long as you are aware of the limitations during usage, you can use AI perfectly fine as a tool.”

The Limitations of Generative AI in Exploratory Testing

If we are going to use AI in ET, we must be cautious. Although Generative AI tools, such as ChatGPT, possess many capabilities, they are not able to understand deep context in the way we humans can. They lack a profound understanding of the product’s purpose, target audience, and expected use by that audience. This is simply because the tool’s training data does not encompass the specific software you wish to test. As long as you are aware of the limitations during usage, you can use AI as a tool.

Generative AI relies on an extensive training dataset as its primary source and is determined by many parameters, which serve as a repository for the collected ‘knowledge.’ Consequently, for example, ChatGPT excels at tasks such as text classification, answering questions, summarizing documents, rewriting documents, and generating text.

Rightful concerns are being raised from various sides about:

• Confusing and incorrectly combining facts and references.
• Relying on sources that are not verified for reliability.
• Bias in generated content.
• Limited understanding of context.

You must therefore take this into account.

How to Ensure a Reliable Result?

You cannot exclusively rely on Generative AI for ET, let alone replace a tester with it. For ET, substantive knowledge, deep product context, product risks, user wishes and behavior, and several other parameters are important. You determine strategies in advance on how to approach the testing. During execution, you make choices based on all those relevant parameters. This is such a complex process that you will miss things when using a Generative AI tool. Consequently, you may overlook risks and errors in the software, with all the resulting consequences when this software ultimately goes to the users.

Examples

I use ChatGPT in this blog, but you are naturally free to choose another Generative AI tool. Below are three examples.

1. Starting with a Basic Strategy for ET To help/inspire your approach to ET, for example, when you want to look at the user-friendliness of a test object, you can ask ChatGPT to generate a test strategy. You will then have a starting point, and from there, you can expand your strategy.

Quick Example ‘Generate a test strategy for exploratory testing when checking whether an application is user-friendly.

ChatGPT

Here is a test strategy for exploratory testing when checking whether an application is user-friendly:

1. Explore the Application: Begin by exploring the application to get a general understanding of the user interface, navigation, and available functions.
2. Identify Key User Flows: Determine the most important tasks that users will perform in the application, such as registering, logging in, searching for products or information, adding items to shopping carts, performing a payment, etc.
3. Create Personas: Establish personas that represent the different types of users of the application, such as new users, returning users, technically skilled users, and non-technical users.
4. User Interface (UI) Tests:
   • Assess the general layout and design of the application.
   • Check if the interface is intuitive and easy to understand for users from different backgrounds.
   • Test the consistency of elements such as colours, fonts, and button styles across the entire application.
5. Navigation Tests:
   • Test the navigation throughout the application to ensure users can easily find what they are looking for.
   • Check if the menu structure is logical and if there are clear call-to-action buttons for important actions.
6. Responsiveness Tests:
   • Test the application on various devices and screen sizes to ensure the layout and functionality remain consistent.
   • Check if the application performs well on both desktop and mobile devices.
7. Usability Tests:
   • Perform user tests with real users to gather feedback on the application’s user-friendliness.
   • Observe how users use the application and note where they encounter problems or difficulties.
8. Accessibility Tests:
   • Check if the application meets accessibility standards, such as the use of alt-text for images, providing keyboard navigation, and avoiding colours that are difficult for colourblind users to distinguish.

2. Generating Fictitious Test Data Based on Parameters

During test execution, you often need (a lot of) test data, and generating it can be a time-consuming task. You also want to avoid using the same data repeatedly out of convenience. You can use ChatGPT to generate fictitious, representative data.

Example ‘Provide me with 1 customer named after a famous cyclist, including a fictitious address, Dutch postal code, a valid IBAN bank account number at a random Dutch bank, date of birth, a car license plate, car make and model, VIN number, number of claim-free years, and type of insurance and an email address, who is a customer of an insurance company and has car insurance.’

ChatGPT

Name: Chris Froome

• Address: Klimweg 10 (Climbing Road 10)
• Postal Code: 5678 KL
• IBAN Bank Account Number: NL34ABNA5678901234 (ABN AMRO)
• Date of Birth: 20-05-1985
• License Plate: 56-KL-MN
• Car: Mercedes-Benz C-Klasse
• Make and Type: Mercedes-Benz C-Klasse
• VIN-number: WDD2050501R123456
• Number of Claim-Free Years: 7
• Type of Insurance: Third-party plus limited casco (WA beperkt casco)
• Email Address: chris.froome@example.com

3. Inspiration for Test Scenarios

Suppose you want to test new functionality in the software and you need a starting point for test scenarios. You can ask if there are subjects you can pay attention to/focus on. Just like with the test strategy question, this can be a great starting point for further development.

Example ‘Create test scenarios to verify the validity of customer data provided for a car insurance policy.’

ChatGPT

Here are some test scenarios to verify the validity of customer data provided for a car insurance policy:

1. Validity of the IBAN Bank Account Number:
   • Check if the IBAN bank account number has a valid format according to the IBAN standard.
   • Check if the bank account number corresponds to a bank in the Netherlands.
2. Validity of the Date of Birth:
   • Check if the date of birth falls within an acceptable range (for example, between 18 and 100 years old).
   • Check that the date of birth is not in the future.
3. Validity of the Car License Plate:
   • Check if the license plate is a valid Dutch license plate.
   • Check if the license plate is unique and not already in use for another car.
4. Validity of the Car Make and Type:
   • Check if the specified make and type correspond to the registered information from the RDW (Netherlands Vehicle Authority).
5. Validity of the VIN number:
   • Check if the VIN number has a valid format according to the standard.
   • Check if the VIN number is unique and not already in use for another car.
6. Number of Claim-Free Years:
   • Check if the provided number of claim-free years falls within a realistic range (for example, between 0 and 99 years).
   • Check that the number of claim-free years is not negative.
7. Type of Insurance:
   • Check if the provided insurance type corresponds to the available options (e.g., Third-Party Liability (WA), WA limited casco, All-Risk).
8. Validity of the Email Address:
   • Check if the email address has a valid format.
   • Check if the email address is unique and not already in use for another customer.
9. Validity of the Address:
   • Check if the provided address exists and is spelled correctly.
   • Check if the postal code corresponds to the provided address.

By executing these test scenarios, it can be guaranteed that the customer data provided for a car insurance policy is valid and correct, which is essential for accurate and reliable processing of the insurance.

Improving the Quality of Generative AI Responses

To get the best possible answers in the future, you need to be clear about your intention. Therefore, pay attention to the following:

• Be highly relevant in the question asked
• Provide accurate and factual information
• Indicate the desired level of detail
• Indicate the desired layout

The three steps you can take to improve the answers from a Generative AI tool are:

1. Add specific context of your application
2. Provide examples of the desired answers
3. Verify the answers

The Art of Exploration

To conclude after this brief analysis that AI is a blessing for ET is perhaps an overstatement, but it is certainly not a curse. Generative AI tools can be helpful aids to challenge yourself further, generate test data, take over simple/boring tasks, and assist you in brainstorming sessions. Understanding deep context is something that Generative AI still truly lacks compared to a tester. By training the Generative AI tool, the answer may become more reliable in the future.

Until then, humans are still better and necessary for critical thinking! Luckily so, because letting your creativity run free is, for me, one of the most enjoyable aspects of the testing profession. Therefore, for now, use Generative AI primarily to kickstart your work and creatively challenge yourself, and not for executing uncontrolled tasks.