CASE STUDY ON GUIDED EXPLORATORY TESTING:

Brief Background: The client is a start up, which provides community portal for the American buyers to express their views and opinion via reviews for various products. The portal slowly evolved and is currently sold as a product to varied retail chains both as a ‘white labelled ‘and customised solution according to the customer needs. Our client has 3 major customers with over 7 websites maintained for 6 browsers combinations and 2 Operating systems.

Current Development Process: The client initially started with a small development team and has grown to 80 developers in around 3.5 years. They currently have a two week iteration in which the first 6 days developers code and in next 4 days they themselves test the application. As the customer base for the client has been growing it became essential to establish a QA practise for providing the safety net. Moreover the client has to cater to the demands of customisation hence they need to optimise the utility of their developers for coding and free them from testing. Client approached us to set up a reliable QA practise.

Problem statement

• A legacy application of 3.5 years which has never been tested in a rigorous fashion.
• As most of the initial designers have left the organisation therefore it is very difficult to understand the business context of available features as well as no documentation is available.
• There is an urgent need to provide test coverage and optimise the entire testing process so that we can free the developers time and contribute to the overall productivity.
• We do not have the exact start and end point to ensure all major business flows are covered.

Our approach:

Given the above constraints, we needed to devise a plan to provide a comprehensive solution in a phased manner.

1. Prioritization: The first step was to prioritise the various features in importance of business value. For e.g. Login and Review modules were given preference over blogs and polls module.This decision was important as we needed to quickly cover the core business areas and provide valuable feedback which had greater significance over others.
2. Breadth over Depth: There were two possibilities in which we could have directed the entire testing process. The first was to cover in depth a particular module before venturing to other modules .The second was to provide basic safety net across various modules and then dive deeper to enhance the test coverage over the period of time.We preferred second approach over the first as our primary intention was to get used to application as soon as possible and come up with  at least smoke test cases across the modules for upcoming releases.
3. Exploratory Testing: The best approach to handle these conditions was to explore the application, understand the functionalities and write test cases simultaneously.The biggest advantage with this approach was that we were able to find bugs from the very beginning and hence were able to contribute while learning.The second advantage was that we were able to come up with various business questions real fast and got them clarified, which highlighted our holistic approach, covering both functional and business areas.
4. Structuring the Exploratory Testing: After few weeks of writing test scenarios  we felt that via exploratory testing we were able to come with smoke scenarios but there were lot of grey areas which were missed out and need to be covered in regression.Moreover this approach does not have a definitive starting and end points hence it cannot be quantified nor can it be time bounded.In order to overcome these loop holes it became important for us to restructure our approach. We wanted to hares the advantages which exploratory testing provides as well as overcome its disadvantages at the same time. The viable solution lied in combining basic flow diagrams with exploratory testing and hence structuring the entire process.

Let me take an example of Review Module:

When we drew the basic flow diagram of this module it appeared like this.

We took these flows as our base and explored around them to come up with exhaustive test coverage.

Advantages of combing Basic Flow Diagram with ET:

• It revealed the test flows which we missed out while doing exploratory testing.
• It provided confidence that all possible flows for a particular module are covered.
• We were able to come with Start Point and End point for each module and hence could quantify the exploratory testing in terms of test flows covered.
• We were able to time bound our effort and come with more accurate effort estimation for writing the test cases.
• We were still able to maintain the pace while widening the test coverage.

Incorporating the Customization: Few minor customizations sometimes add an extra flow or modify an existing flow as well as there are few customised versions of the functionalities but the basic flow of the modules remains the same.

Since we were able to visualise these flows with help of the flow diagrams it became easy for us to be become more flexible and robust and we easily incorporated these changes in the current test scenarios.

Other added Advantages:

• We were able to build a comprehensive regression suite within 4 weeks using 2 resources consisting of around 240 scenarios.
• We were able to log around 40 bugs which proved to be a valuable feedback for the client to modify the current code base.
• We were able to optimise the entire regression cycle from initial period of 4 days to 2.5 days and smoke tests from 3.5 hrs to 2 hrs.

Key Business Value delivered:

• The client was able to see the immediate benefits in terms of bugs, which provided effective feedback to incorporate changes in code base.
• The client developers need not be responsible for testing after 6 weeks of overlap.
• The test scenarios identified are used by the automation team to build automation scripts for CI.

Conclusion:

1. Exploratory testing is an effective way for providing quick feedbacks for legacy applications with little or no functional knowledge.
2. Exploratory testing can help us to be productive and learn at the same time hence adding value.
3. Exploratory testing when coupled with Basic flow diagrams becomes an effective tool to provide extensive test coverage.
4. Structured exploratory testing provides a definitive start and end point and hence can be tracked and time bounded.