Cricket and Software Testing - an Analogy! Howzat!

Yesterday I was reading this news (4 obvious errors show inadequate concentration), when the idea of this post struck my mind. To be honest, the mention of the words “errors” and “concentration” in the news headline caught my attention and when I finished reading the news item, I realized that it was not related with any software errors as I had anticipated before starting reading it. However, on a second thought, when I tried to correlate it with software testing and software testers, it started making some sense to me and the following paragraphs are a result of it.

Before I continue further, let me make it clear that when I say cricket, I am not talking about the insect cricket (Class: Insecta, Family: Gryllidae). Rather, I am talking about the sport cricket (Cricket is a bat-and-ball sport contested by two teams, usually of eleven players each). Let us see if we can correlate the sports cricket with that of software testing! I clearly understand that I might sound awfully stupid when I say so and even try to correlate two things as diverse as that of cricket and software testing. But if you are also an ardent fan of cricket as I am and know something about the sport, then I guess, you would not fail to appreciate the attempt after finishing reading this post. In case, you are not a cricket fan, you can still continue reading this post till the end to see what I have to offer via this write-up! I am going to identify and list out few points that seem similar between software testing and cricket:

1. (Un)Predictability: Experts say, ‘unpredictability of cricket is its greatest charm’! It is very difficult to predict a win or loss before the last ball is bowled. And it is the unpredictability, which makes cricket so interesting. Teams might look strong or weak on a piece of pen and paper. But in cricket, unpredictability reins, not the statistics.

Coming to software testing, there could be testers who believe that it is easy and straightforward to expect some fixed output after executing a series of test steps, but I only wish; testing software was as simple as that! Echoing some experts in software testing: trying to predict the result of a test in terms of PASS or FAIL criteria can be one of those dangerous traps in software testing world, where a tester can shoot at his own feet! Irrespective of the number of test scripts (either manual test cases or automated test scripts) a tester has written, until the tester gets the application module to test, nothing can be told for sure about the state of the application and its behavior. Unpredictability is one of those things that make software testing so much fun.

2. Skills: Cricket is a game where only skillful players can make their team a winner. I am not sure about other countries, but if you have ever been to India, chances are high that you might have seen cricket being played on a street behind your hotel room! They say, cricket is a fever here in India. Here people are so obsessed with the game that they not only play and watch cricket but also eat, sleep and even drink cricket! I have played fair amount of cricket in my school and college days. You might have too. But the reason, why players like Sachin Tendulkar, Brian Lara and Ricky Ponting are considered as one of the finest cricketers and not us, lies in the cricketing skills they possess.

Likewise, in software testing too, it is the testing skill that differentiates a good tester from a mediocre one. The one, who has the better testing skills in his arsenal, can find more important defects quickly in the software that he is testing. Learning, practicing and applying are three golden rules to acquire any skill. With determination and strong will power, nothing is impossible to learn. Fortunately, software testing is no exception!

3. Game Planning (Knowing the Opponent): Professional cricket is all about knowing the strengths and weaknesses of the opponent team and devising a game plan in order to combat their strengths and to exploit their weaknesses.

In software testing, knowing the testing mission is the first step in determining the goal of the testing effort. Without being clear about the goal, it might turn out fatal to go about testing straightaway. Once the tester is clear about the testing mission, then he can analyze his chances of success or failure depending on the availability and expertise of the resource in hand and the complexity of the testing problem. e.g. imagine a situation where the tester has to test the application to find out it’s robustness to guard against hackers and other malicious users. Knowledge of a fair deal of details about the level of attack that can be attempted against the application, can give the tester a better chance to plan out a strategy to emulate the attack and to test how the application guards against it.

4. Handling Pressure: Winning a game of cricket is all about handling the pressure well. The team that is able to handle the tremendous amount of pressure of the game finally wins the game.

For those who think or have been told that software testing is an easy job and can be done by every Tom Dick and Harry, let me warn you, you have been terribly misguided! Testing is a career, which demands lot of intellectuality and stability of mind to work under variety of pressures (like technical pressure, pressure due to workload, managerial pressure, pressure of deadline, pressure arising from the nature of the job etc). As testers, the basic requirement of our job demands us to deliver bad news (presence of defects, buggy modules that fail miserably on testability grounds etc) to different stakeholders (the programmers, management staffs, clients) of the product under development. Nobody likes to hear bad news. Unfortunately, gone are the days when messengers were not hanged just because they brought in a bad news to the king! Hence, unless the tester is quite good at handling the pressure arising as a byproduct of his work, and is not so good at being diplomatic, he might find it tough to carry on with his job for long. On the other hand, the tester who has got the capability to handle the pressure till the end, has every chance of winning the Testing World Cup!

5. Adaptability: If you have been watching cricket for sometime now, then you must be experienced enough to know that whenever a team visits a different continent for playing cricket, it often finds it difficult to play up to its usual standards, of course unless we are talking about a team like Australia. Most Asian teams find it difficult to play down under and the vice versa. And the reason lies mainly in the difference in the pitch condition in the different continents. Soil texture, clay quality, humidity, temperature, amount of grass, dust etc can affect the behavior of a cricket pitch. The players who can adapt themselves quickly with the new environment can gain an edge in the game.

Just as no two cricket pitches can be exactly identical, no two AUTs (Application Under Test) can be same. Hence, I often get surprised when I see testers trying to clean themselves off testing assignments with excuses such as:

» “I have not tested anything like this before!”
» “This platform is completely new to me!”
» “I am new to this technology. I can not test it!”
» “I am a manual tester. I can not use this test automation tool!”
» “I need the domain knowledge before I can test this application!”
» “Hey, you don’t have any base documents (URS, SRS, BRS blah blah…)! How can I start testing your application?”

To me, this simply translates as a bad attitude of a tester, who is not ready to adapt and more importantly not ready to learn something new. For such testers, I have just one thing to say:

"Don’t forget that, even big and powerful animals like the Dinosaurs and the Wooly Mammoth became extinct just because they FAILED to adapt to their changing surrounding environments; we are just testers!"

When I say so, I DON’T mean that I am a great tester who has been excellent at adaptability throughout his testing career. On the contrary, I have been through several such instances when I was stupid enough to oppose such changes and was in fact, hostile to proposed changes. And don't be too surprised when I tell you that, some of the excuses that I have mentioned above were from my own mouth! However, I have learnt my lesson of adaptability in the hard, bitter way. So I just wanted to warn others who are new and passionate about testing but might commit the same mistake of lack of adaptability that I have committed in my earlier years of testing career.

6. Patience: Cricket is a game of patience. At times, a batsman might be finding it tough to steal a run due to the sharp bowling attack, tough ground conditions etc. But if the batsman has got enough temperament and patience, he might soon find it easy to not only steal a single but also score boundaries against the same bowler under same pitch conditions. Similarly with patience and consistent good bowling, a bowler could turn around the fate of a match by picking up quick wickets.

Similarly, a lot of patience is required while testing an application. There might be days when nothing seemed to work right for you and you would end up without catching a single defect in the whole testing session. There might be times when you would find yourself hitting your head on the keyboard while trying to find ways to reproduce a hard-to-reproduce defect. But you ask any expert in software testing, and he would suggest you to have patience. Putting in extra effort and having patience can help you come out of such dry days of testing.

7. Team Game: Cricket is a team game. Here the team wins the match, which displays excellent exhibition of team play. And so is software testing. All the team members of a testing team can’t be with equal skill sets and with equal level of expertise. Having a testing team with members with different specialization can add to the versatility of the team’s overall performance. I do understand that there can be testing geniuses, who are jack of all testing trades and are all-rounders when it comes to testing. But finding such a tester can be a tough task for any test manager. So as a workaround a good test manager might look forward to build a team which has testers specializing in different aspects of testing (functionality testing, performance/stress testing, database testing, usability testing, automation engineers, risk-based testing etc).

8. Lack of Concentration can Doom you: Cricket requires a high level of concentration. As a batsman you loose your concentration and you could loose your wicket. As a bowler you loose your concentration and you could be hit for boundaries all over the ground! As a fielder you loose your concentration and you could drop a catch (remember! catches win matches!) or give extra runs by mis-fielding.

Coming back to testing, imagine a tester who misses to identify a defect that just happens in front of his eyes, as he was looking for some other defect at that moment. Experts call this phenomenon as inattentional blindness. This can also happen if your mind is too tired due to continuous testing and you start loosing concentration, and in turn, start loosing defects!

9. Wearing safety Helmets and Testers: Batsmen (and even the close-in fielders) wear helmets to safeguard their head from the fast coming deliveries from the bowler. In case of software testing, unfortunately, the testers are like the safety helmet! They act as the representative of the client/end users of the project. And act as the last line of defense of the client/end users against the possible ways in which the software might fail. So as testers, start feeling proud of your profession, as you are safeguarding somebody by taking the hits yourself. By spending time with all those irritating defects while testing, you are making sure that your end user does not have to deal with them in future.

10. Learning Lessons from Failures: Even world champions like Australia was once defeated in a cricket match against the under-dogs like Bangladesh. Success and failures are two integral part of any game. The team that wins today can loose the next match and vice versa. But what differentiates a world champion from an average team is the extent to which they learn their lessons from their failures.

On a similar note, a software testing project can go terribly bad due to various reasons. I have witnessed and have been part of few such testing projects that went nowhere nearer where they were intended to go! Some were complete disasters and the whole project development had to be shut down. I am sure, if you are in testing field for sometime now, then you also might have seen, experienced, heard about such failures. But each experience, good or bad, has some lessons in it. A good tester tries to learn lessons from his past failures to make sure they are not repeated in future. The failure might not have been as a result of a mistake on the part of the tester, but still, there might be some lessons in that failure that could help the tester in becoming better at his testing in future.

If you have reached this paragraph after patiently reading through my post, then I must appreciate you for your patience. You have certainly shown a desirable skill of a good tester – patience. :) I would like to hear your ideas about my analogy between cricket and software testing. Can you think of more such points, which can link them together? Let me know your ideas by commenting. Howzat!!!

Happy Testing…

Article by : Debasis

Read more...

Five steps to revive the rejected bug - Dream of the Tester

Testing by itself does not improve software quality. Test results are an indicator of quality, but in and of themselves, they don't improve it. Trying to improve Software quality by increasing the amount of testing is like trying to lose weight by weighing yourself more often. What you eat before you step onto the scale determines how much you will weigh, and the software development techniques you use determine how many errors testing will find. If you want to lose weight, don't buy a new scale; change your diet. If you want to improve your software, don't test more; develop better. A tester reports a software bug/defect in the application he is testing. He feels that this is a genuine defect that needs attention. But to his shock and astonishment he finds out that his bug is rejected by the developer team with an excuse of the-application-works-as-designed ! This can happen with any tester at some point in his testing career. And this can be quite frustrating too; especially if the tester feels that the software defect is a serious one and has got potential to cause severe damage to the client/end user if the software is shipped with the defect.

Having said that, not every defect that is rejected is worth fighting for. So as the first step of self-assessment, a tester might go back to the defect report that he had submitted to the programmers and verify if the defect report was well defined! Few things worth verifying in the submitted defect report are: If the defect report had a well-written summary line and the steps mentioned were readable (clear and precise). Use of words might play a vital role in deciding the fate of a defect report. A single ambiguous word might suppress the seriousness of the defect and your defect report could look like a bunch of garbage, wasting the bandwidth of the defect tracking system! If the defect report contained any unnecessary step(s), adding to the confusion. If the report clearly mentioned what actually happened and what you expected to happen.

If you had mentioned the consequences of the defect, in case it is allowed to slip through the Release Phase. If your tone of voice sounded sarcastic, threatening or careless in the defect report. Was it in a way, capable of making the programmer confused, unreceptive, or angry?A well-written defect report can differentiate a best-selling bug from a flop show! If you had missed to report the defect properly, you should not blame the programmer for turning down your defect as "rejected";! May be you should spend some time on your bug/defect reporting skills and try once again. But suppose, you had reported the defect quite neatly and still it was rejected, decide if you would choose to go with the decision or rather appeal against it as you still strongly feel that this is a serious defect that needs immediate attention. In case, you are planning to appeal against the rejection these are few things that you might consider doing in order to increase your chance of success:

1. Patch the holes : Look for loopholes in your original defect report that could be supported with further investigative data to strengthen your case. When you are going for an appeal, you should anticipate attacks on the weaker areas of your original report. You should understand that your report was weak and unpersuasive at the first place. So try and gather as mush information to make it appealing this time around.

2. Follow it up : Do some additional follow up testing around the original defect in an attempt to find more serious consequences. If you are able to find more areas that are affected by the defect and more severe consequences, it should add to your confidence level. A defect that infests a wider range of functionalities and has severe consequences has more chance of getting attention.

3. Follow the Money : There is a popular doctrine in criminal investigation; “in most of the crime cases, if you will follow the money you should soon able to reach the criminal! Same can be applied in testing too, while appealing against rejection of a defect. Talk to the major stakeholders like the Managers, the Client, Sales department staffs, Technical Support team, and even the Technical Writers. Try to find out who will be most affected if the Product is shipped along with the defect. Try to get an idea of the financial loss that can result due to this defect if left unfixed. As James Bach defines "A bug/defect is something that bugs someone who matters?! Try to identify the "someone" for whom your defect really matters and find out how costly it matters.

4. Build a Scenario to support your Testing : It's time for story telling. This is where a tester’s story telling capability comes in handy. Use your imagination and your creativity to weave around a realistic story that sounds appealing and at the same time is capable of conveying the seriousness of the rejected defect. Build some scenarios that exemplify how a real-time user might come across the defect and how it might affect the user in a severe way.

5. Look out for similar defects in Competitors : Take advantage of the immense knowledge base of the Internet to find out some case where one of your competitors had released their Product with a defect similar to yours and had to face terrible consequences. Check in the recent press releases, forum discussions, news reports, courtroom cases for a similar case where a defect (similar to yours) had caused serious loss (financial loss in terms of loss in revenue, loss in credibility, loss in loyal customers etc) to a competitor. If you already take notes of important events related to testing, also look into your moleskine notebook for any similar incident that you might have recorded in there! If you are lucky enough to find such a case, your appeal should sound lot better in the review meeting!

With proper planning, you can increase the chance of the revival of your rejected defect in the review meeting. Do your homework well and build your case more effectively. But at any cost, never appeal for a defect that you yourself are not convinced with! If your further investigation reveals that this defect is not worth fighting for and has less chance of affecting an end user in real life scenario, better you should think twice before going for an appeal. Fighting for a defect that is not appealing to you, might result in waste of time and loss of your credibility as a software tester! Any thoughts?

Read more...

What is the difference between QA and QC???

What is Software Quality?

Though there are a number of definitions propounded by the gurus in the field - each having its own adherents, two definitions that are widely accepted and that are complementary to each other are:

- Conformance to explicitly stated functional and performance requirements, explicitly documented development standards, and implicit characteristics that are expected of all professionally developed software.

- The degree to which a system, component, or process meets specified requirements and customer or user needs or expectations.

Quality assurance and quality control both contribute in delivering a high quality software product though the way they go about it is different. This can be illustrated by looking at the definitions of the two.

What is Software Quality Assurance?

Software QA involves the entire software development PROCESS - monitoring and improving the process, making sure that any agreed-upon standards and procedures are followed, and ensuring that problems are found and dealt with. It is oriented to ’prevention’.

This is a ’staff’ function, and is responsible for establishing standards and procedures to prevent defects and breakdowns in the SDLC. The focus of QA is prevention, processes, and continuous improvement of these processes.

What is Software Quality Control?

This is a department function, which compares the standards to the product, and takes action when non-conformance is detected for example testing.

This involves operation of a system or application under controlled conditions and evaluating the results (e.g., ’if the user is in interface A of the application while using hardware B, and does C, then D should happen’). The controlled conditions should include both normal and abnormal conditions. Testing should intentionally attempt to make things go wrong to determine if things happen when they shouldn’t or things don’t happen when they should. It is oriented to ’detection’.

Relationship between QC and QA

An application that meets its requirements totally can be said to exhibit quality. Quality is not based on a subjective assessment but rather on a clearly demonstrable, and measurable, basis. Quality.

- Quality Control is a process directed at validating that a specific deliverable meets standards, is error free, and is the best deliverable that can be produced. It is a responsibility internal to the team.

- QA, on the other hand, is a review with a goal of improving the process as well as the deliverable. QA is often an external process. QA is an effective approach to producing a high quality product.

One aspect is the process of objectively reviewing project deliverables and the processes that produce them (including testing), to identify defects, and then making recommendations for improvement based on the reviews. The end result is the assurance that the system and application is of high quality, and that the process is working. The achievement of quality goals is well within reach when organizational strategies are used in the testing process. From the client's perspective, an application's quality is high if it meets their expectations.

Read more...

Difference between Load Testing and Stress Testing

One of the most common misuse of terminology is treating "Load Testing" and "Stress Testing" as synonymous. The consequence of this ignorant semantic abuse is usually that the system is neither properly "load tested" nor subjected to a meaningful stress test.

1. Stress testing is subjecting a system to an unreasonable load while denying it the resources (e.g., RAM, disc, mips, interrupts, etc.) needed to process that load. The idea is to stress a system to the breaking point in order to find bugs that will make that break potentially harmful. The system is not expected to process the overload without adequate resources, but to behave (e.g., fail) in a decent manner (e.g., not corrupting or losing data). Bugs and failure modes discovered under stress testing may or may not be repaired depending on the application, the failure mode, consequences, etc. The load (incoming transaction stream) in stress testing is often deliberately distorted so as to force the system into resource depletion.

2. Load testing is subjecting a system to a statistically representative (usually) load. The two main reasons for using such loads is in support of software reliability testing and in performance testing. The term "load testing" by itself is too vague and imprecise to warrant use. For example, do you mean representative "load", "overload", "high load" etc. In performance testing, load is varied from a minimum (zero) to the maximum level the system can sustain without running out of resources or having, transactions suffer (application-specific) excessive delay.

3. A third use of the term is as a test whose objective is to determine the maximum sustainable load the system can handle.
In this usage, "load testing" is merely testing at the highest transaction arrival rate in performance testing.

Read more...

When to Start and Stop Testing????????

Testing is sometimes incorrectly thought as an after-the-fact activity; performed after programming is done for a product. Instead, testing should be performed at every development stage of the product. Test data sets must be derived and their correctness and consistency should be monitored throughout the development process.

If we divide the lifecycle of software development into “Requirements Analysis”, “Design”, “Programming/Construction” and “Operation and Maintenance”, then testing should accompany each of the above phases. If testing is isolated as a single phase late in the cycle, errors in the problem statement or design may incur exorbitant costs. Not only must the original error be corrected, but the entire structure built upon it must also be changed. Therefore, testing should not be isolated as an inspection activity. Rather testing should be involved throughout the SDLC in order to bring out a quality product.

Testing Activities in Each Phase
The following testing activities should be performed during the phases
1. Requirements Analysis
- Determine correctness
- Generate functional test data.
2. Design
- Determine correctness and consistency
- Generate structural and functional test data.
3. Programming/Construction
- Determine correctness and consistency
- Generate structural and functional test data
- Apply test data
- Refine test data.
4. Operation and Maintenance
- Retest.

Details of testing in all phases:

1. Requirements Analysis
The following test activities should be performed during this stage:

1.1 Invest in analysis at the beginning of the project - Having a clear, concise and formal statement of the requirements facilitates programming, communication, error analysis an d test data generation.
The requirements statement should record the following information and decisions:
a. Program function - What the program must do?
b. The form, format, data types and units for input.
c. The form, format, data types and units for output.
d. How exceptions, errors and deviations are to be handled.
e. For scientific computations, the numerical method or at least the required accuracy of the solution.
f. The hardware/software environment required or assumed (e.g. the machine, the operating system, and the implementation language).

Deciding the above issues is one of the activities related to testing that should be performed during this stage.

1.2 Start developing the test set at the requirements analysis phase - Data should be generated that can be used to determine whether the requirements have been met. To do this, the input domain should be partitioned into classes of values that the program will treat in a similar manner and for each class a representative element should be included in the test data.
In addition, following should also be included in the data set:
(1) boundary values
(2) any non-extreme input values that would require special handling.

The output domain should be treated similarly.
Invalid input requires the same analysis as valid input.

1.3 The correctness, consistency and completeness of the requirements should also be analyzed - Consider whether the correct problem is being solved, check for conflicts and inconsistencies among the requirements and consider the possibility of missing cases.

2. Design
The design document aids in programming, communication, and error analysis and test data generation. The requirements statement and the design document should together give the problem and the organization of the solution i.e. what the program will do and how it will be done.

The design document should contain:
· Principal data structures.
· Functions, algorithms, heuristics or special techniques used for processing.
· The program organization, how it will be modularized and categorized into external and internal interfaces.
· Any additional information.

Here the testing activities should consist of:
- Analysis of design to check its completeness and consistency - the total process should be analyzed to determine that no steps or special cases have been overlooked. Internal interfaces, I/O handling and data structures should specially be checked for inconsistencies.

- Analysis of design to check whether it satisfies the requirements - check whether both requirements and design document contain the same form, format, units used for input and output and also that all functions listed in the requirement document have been included in the design document. Selected test data which is generated during the requirements analysis phase should be manually simulated to determine whether the design will yield the expected values.

- Generation of test data based on the design - The tests generated should cover the structure as well as the internal functions of the design like the data structures, algorithm, functions, heuristics and general program structure etc. Standard extreme and special values should be included and expected output should be recorded in the test data.

- Re-examination and refinement of the test data set generated at the requirements analysis phase.

The first two steps should also be performed by some colleague and not only the designer/developer.

3. Programming/Construction
Here the main testing points are:

- Check the code for consistency with design - the areas to check include modular structure, module interfaces, data structures, functions, algorithms and I/O handling.

- Perform the Testing process in an organized and systematic manner with test runs dated, annotated and saved. A plan or schedule can be used as a checklist to help the programmer organize testing efforts. If errors are found and changes made to the program, all tests involving the erroneous segment (including those which resulted in success previously) must be rerun and recorded.

- Asks some colleague for assistance - Some independent party, other than the programmer of the specific part of the code, should analyze the development product at each phase. The programmer should explain the product to the party who will then question the logic and search for errors with a checklist to guide the search. This is needed to locate errors the programmer has overlooked.

- Use available tools - the programmer should be familiar with various compilers and interpreters available on the system for the implementation language being used because they differ in their error analysis and code generation capabilities.

- Apply Stress to the Program - Testing should exercise and stress the program structure, the data structures, the internal functions and the externally visible functions or functionality. Both valid and invalid data should be included in the test set.

- Test one at a time - Pieces of code, individual modules and small collections of modules should be exercised separately before they are integrated into the total program, one by one. Errors are easier to isolate when the no. of potential interactions should be kept small. Instrumentation-insertion of some code into the program solely to measure various program characteristics – can be useful here. A tester should perform array bound checks, check loop control variables, determine whether key data values are within permissible ranges, trace program execution, and count the no. of times a group of statements is executed.

- Measure testing coverage/When should testing stop? - If errors are still found every time the program is executed, testing should continue. Because errors tend to cluster, modules appearing particularly error-prone require special scrutiny.
The metrics used to measure testing thoroughness include statement testing (whether each statement in the program has been executed at least once), branch testing (whether each exit from each branch has been executed at least once) and path testing (whether all logical paths, which may involve repeated execution of various segments, have been executed at least once). Statement testing is the coverage metric most frequently used as it is relatively simple to implement.
The amount of testing depends on the cost of an error. Critical programs or functions require more thorough testing than the less significant functions.

4. Operations and maintenance
Corrections, modifications and extensions are bound to occur even for small programs and testing is required every time there is a change. Testing during maintenance is termed regression testing. The test set, the test plan, and the test results for the original program should exist. Modifications must be made to accommodate the program changes, and then all portions of the program affected by the modifications must be re-tested. After regression testing is complete, the program and test documentation must be updated to reflect the changes.


When to stop testing?
"When to stop testing" is one of the most difficult questions to a test engineer. The following are few of the common Test Stop criteria:

- All the high priority bugs are fixed.
- The rate at which bugs are found is too small.
- The testing budget is exhausted.
- The project duration is completed.
- The risk in the project is under acceptable limit.

Practically, the decision of stopping testing is based on the level of the risk acceptable to the management. As testing is a never ending process, we can never assume that 100 % testing has been done, we can only minimize the risk of shipping the product to client with X testing done. The risk can be measured by Risk analysis but for small duration / low budget / low resources project, risk can be deduced by simply: -

- Measuring Test Coverage.
- Number of test cycles.
- Number of high priority bugs.

Read more...