Join the social network of Tech Nerds, increase skill rank, get work, manage projects...
  • Exploratory Automation Testing: To Use or Not to Use - An In-depth Analysis

    • 0
    • 0
    • 0
    • 0
    • 0
    • 0
    • 0
    • 0
    • 627
    Comment on it


    Agile development has evolved quickly over the last 20 years, providing a fundamental shift in how products can be delivered to users. It is based on the concepts of collaboration, flexibility, and transparency for the entire development process.


    Many companies - including Microsoft and Google - have embraced Agile methodology for their software development. Scrum, Kanban, and XP are the most popular Agile frameworks, all having similar principles and end results along with the ability to meet the needs of users and customers quickly and easily.


    In this regard, it’s quite surprising that the way in which testing is delivered is not always Agile. Usually, the activities of testing can be divided into the following steps:


    Requirements clarification > Test design > Testing > Results analysis


    The main advantage of the proposed approach is transparency. The whole team is very much aware of the tasks assigned, while managers and customers always know exactly what is being done.


    However, such methodology poses certain challenges:


    1. The testing team need a lot of time to work on detailed documents such as test cases and test scenarios.

    2. The test documentation quickly becomes outdated.

    3. The team spends significant time and effort renewing documentation.

    4. After running a test case 2 or 3 times, some bugs cannot be found immediately. To find new bugs, you have to create new test cases.

    5. Ad-hoc testing, which is performed after regression testing if there’s time, provides better results than using test cases.


    Obviously, these prospects are not particularly encouraging. This is why it’s vital to make testing as agile as development. But just how exactly do you do this? Exploratory testing (ET) definitely seems like a good option.


    The most well known definition of this notion was coined by Cem Kaner:


    "A style of software testing that emphasizes the personal freedom and responsibility of the individual tester to continually optimize the quality of his/her work by treating test-related learning, test design, test execution, and test result interpretation as mutually supportive activities that run in parallel throughout the project."


    It seems to be a rather simple approach – we come up with some tests, immediately apply them to the app, analyze the results, then create new tests, etc. But Cem Kaner and other followers of the approach emphasize that it is impossible to carry out exploratory testing without a deep understanding of different testing techniques.


    Positive and negative, CRUD and pairwise testing, Boundary Value Analysis, cross-browser, end-to-end, and role-based testing . . . the list of examples is unending. So, the more rich skill set you possess, the more diverse your tests will be and the more bugs you will find.


    If you want to adopt an exploratory testing mindset, here are the core questions to consider before implementation:


    - Is it possible to replace scripted testing with an exploratory testing approach?

    - How can you make the exploratory testing process more transparent to a customer?

    - How do you organize exploratory testing for your team so that engineers don’t conduct the same feature tests?


    Let’s start with the last question. To find the answer, you might want to pick up a book by James Whittaker called Exploratory Software Testing. In this book, the renowned software testing expert introduces a powerful exploratory technique for finding bugs and demonstrates how to use it successfully. Interestingly enough, he suggests following the tour metaphor when speaking about testing.


    But what does he mean by that? Whittaker uses the term touring tests to refer to test styles that any engineer has used before.


    He draws an interesting analogy with traveling in stressing that travelers - like QA engineers - always have limited time and many places to visit. They always try to set the route beforehand to visit the places that correspond to the main purpose of the journey.


    Using this metaphor, the author points out that most companies split up testing responsibilities in a way where one test engineer is assigned a feature, and then he tests it from different angles. However, Whittaker's approach is to redefine this process. He suggests giving a test engineer one mission and then letting them test all features from that angle.


    So this approach deals specifically with the problem of overlapping. Even if engineers test the same feature, they do it from different perspectives.


    Let’s discuss the next question: how can you make the testing process more transparent. As James Bach writes in his book Exploratory Testing Explained, there are two ways to control the team tasked with exploratory testing:


    1) Managing by delegation (treat testers as semi-independent creative agents who manage their own time)

    2) Managing by participation (empowering all team members to participate in the testing)


    Both approaches have advantages and disadvantages. The first one presupposes a certain level of trust within the team as there’s no clear evidence that particular features were tested and how it was done. You can only judge by the number of reported bugs and by brief sketches.


    The second method requires a lot of time. You have to postpone all the other tasks to dive into testing. But the question arises, what approach should I choose? It’s totally up to you.


    Finally, let’s discuss the last question: is it possible to replace scripted testing with exploratory testing? The answer will be yes and no.


    So, here are the cases where you can apply exploratory testing and when it’s better not to use it. The projects where exploratory testing will definitely work for you are the following:

    1. Projects with rapid development and frequent releases

    2. Any situation when scripted testing fails to find bugs

    3. When you need to learn the product quickly

    4. When rapid feedback on a feature or product is required

    5. When you need to check the work of another tester

    6. When you need to investigate and isolate a bug


    Also, keep in mind that there are some situations where it’s better not to apply exploratory test execution:

    1. Applications in military, health, insurance and financial domains where the structured approach and double checks will give more confidence that everything is tested and bugs are hardly missed

    2. Projects where focus is devoted to process and documentation. Since ET is almost always not documented, it’s not applicable on such projects.

    3. When testers lack experience


    To use exploratory testing or not is for you to decide. But remember that it is a powerful way to speed up the software testing process. It brings out-of-the-box thinking to QA engineers and encourages them to come up with practical and real-time test cases for finding a defect.


    The success of exploratory testing depends on many factors, including the tester’s skill set and creativity, their experience, ability to set goals correctly, etc. One thing is certain: it’s definitely worth trying to see whether it is an optimal solution for your project.


 0 Comment(s)

Sign In

Sign up using

Forgot Password
Fill out the form below and instructions to reset your password will be emailed to you:
Reset Password
Fill out the form below and reset your password: