Monday, June 18, 2007

Automation Testing versus Manual Testing

Automation Testing versus Manual Testing

I met with my team’s automation experts a few weeks back to get their input on when to automate and when to manually test. The general rule of thumb has always been to use common sense. If you’re only going to run the test one or two times or the test is really expensive to automation, it is most likely a manual test. But then again, what good is saying “use common sense” when you need to come up with deterministic set of guidelines on how and when to automate?

Pros of Automation
  1. If you have to run a set of tests repeatedly, automation is a huge win for you

  2. It gives you the ability to run automation against code that frequently changes to catch regressions in a timely manner

  3. It gives you the ability to run automation in mainstream scenarios to catch regressions in a timely manner (see What is a Nighlty)

  4. Aids in testing a large test matrix (different languages on different OS platforms). Automated tests can be run at the same time on different machines, whereas the manual tests would have to be run sequentially.
Cons of Automation
  1. It costs more to automate. Writing the test cases and writing or configuring the automate framework you’re using costs more initially than running the test manually.

  2. Can’t automate visual references, for example, if you can’t tell the font color via code or the automation tool, it is a manual test.
Pros of Manual
  1. If the test case only runs twice a coding milestone, it most likely should be a manual test. Less cost than automating it.

  2. It allows the tester to perform more ad-hoc (random testing). In my experiences, more bugs are found via ad-hoc than via automation. And, the more time a tester spends playing with the feature, the greater the odds of finding real user bugs.
Cons of Manual
  1. Running tests manually can be very time consuming

  2. Each time there is a new build, the tester must rerun all required tests - which after a while would become very mundane and tiresome.
Other deciding factors
  1. What you automate depends on the tools you use. If the tools have any limitations, those tests are manual.

  2. Is the return on investment worth automating? Is what you get out of automation worth the cost of setting up and supporting the test cases, the automation framework, and the system that runs the test cases?
Criteria for automating
There are two sets of questions to determine whether automation is right for your test case:

Is this test scenario automatable?
  1. Yes, and it will cost a little

  2. Yes, but it will cost a lot

  3. No, it is no possible to automate
How important is this test scenario?
  1. I must absolutely test this scenario whenever possible

  2. I need to test this scenario regularly

  3. I only need to test this scenario once in a while
If you answered #1 to both questions – definitely automate that test
If you answered #1 or #2 to both questions – you should automate that test
If you answered #2 to both questions – you need to consider if it is really worth the investment to automate

What happens if you can’t automate?
Let’s say that you have a test that you absolutely need to run whenever possible, but it isn’t possible to automate. Your options are
  1. Reevaluate – do I really need to run this test this often?

  2. What’s the cost of doing this test manually?

  3. Look for new testing tools

  4. Consider test hooks

===================================

30 comments:

Anonymous,  December 24, 2012 at 10:13 PM  

My programmer is trying to convince me to move to .

net from PHP. I have always disliked the idea because of the costs.

But he's tryiong none the less. I've been using Movable-type on a variety of websites for about a
year and am nervous about switching to another platform.
I have heard very good things about blogengine.net. Is there a way I can import
all my wordpress posts into it? Any help would be really appreciated!



my blog - auf rechnung trotz schufa
Also visit my site ;

Anonymous,  December 27, 2012 at 6:02 AM  

Hi to every one, it's genuinely a pleasant for me to go to see this web page, it contains valuable Information.
Feel free to surf my site : private krankenversicherung rentner

Anonymous,  December 27, 2012 at 6:28 AM  

What's Happening i'm new to this, I stumbled upon this I've found It absolutely useful and it has helped me out loads. I hope to give a contribution & help other customers like its aided me. Great job.
Also visit my web site :: Günstige gesetzliche krankenversicherung vergleich

Anonymous,  December 27, 2012 at 12:08 PM  

Thanks , I have just been looking for info
about this topic for a while and yours is the best I've found out so far. However, what in regards to the bottom line? Are you certain in regards to the source?

Feel free to surf to my blog post ... rechner private krankenversicherung
Also visit my homepage

Anonymous,  December 28, 2012 at 1:30 PM  

Hey there! I know this is kinda off topic but I was wondering which blog platform are you using for
this website? I'm getting fed up of Wordpress because I've had issues
with hackers and I'm looking at options for another platform. I would be great if you could point me in the direction of a good platform.

Check out my web blog: how Does Clickbank work
Feel free to surf my homepage

Anonymous,  December 28, 2012 at 3:42 PM  

I needed to thank you for this wonderful read!
! I absolutely loved every little bit of it. I have you saved as a favorite to look at new
stuff you post…

My weblog - private Krankenkassen preisvergleich
Feel free to surf my site -

Anonymous,  December 28, 2012 at 6:58 PM  

If some one wants to be updated with most recent technologies therefore he must be pay a quick visit this website
and be up to date daily.

my web page:
My webpage ::

Anonymous,  December 29, 2012 at 8:30 PM  

Have you ever considered about adding a little bit more than just your articles?
I mean, what you say is valuable and everything.
However imagine if you added some great pictures or videos to give your posts more, "pop"!
Your content is excellent but with pics and clips, this site could certainly be one
of the best in its field. Very good blog!

Feel free to surf to my page:
My web site ::

Anonymous,  December 30, 2012 at 11:17 PM  

An outstanding share! I have just forwarded this onto a friend who was
conducting a little research on this. And he actually ordered me dinner simply because I found it for him.

.. lol. So allow me to reword this.... Thank YOU for the meal!
! But yeah, thanx for spending time to talk about this issue here on
your site.

My web site ... privat krankenversicherung rechner
Feel free to surf my homepage :

Anonymous,  December 31, 2012 at 2:18 PM  

Informative article, just what I wanted to find.
Review my web blog : free web hosting company

Anonymous,  December 31, 2012 at 6:24 PM  

Thanks for sharing your info. I really appreciate your efforts and I will be waiting for your next post thanks once again.
Have a look at my blog post : sanibel island vacation packages

Anonymous,  January 1, 2013 at 2:49 AM  

Hello there! This post could not be written any better!

Reading through this post reminds me of my old room mate!
He always kept chatting about this. I will forward this post
to him. Pretty sure he will have a good read.

Thanks for sharing!

my web-site sofortkredit für arbeitslose
Here is my homepage

Anonymous,  January 1, 2013 at 3:21 PM  

Greetings! I know this is kinda off topic but I was wondering which blog
platform are you using for this site? I'm getting tired of Wordpress because I've had issues with hackers and I'm looking at alternatives for another platform. I would be awesome if you could point me in the direction of a good platform.

Look at my web blog gesetzliche krankenversicherung vergleichen
My weblog :

Anonymous,  January 1, 2013 at 11:52 PM  

Unquestionably imagine that which you said. Your favorite reason
seemed to be at the web the easiest factor to consider of.
I say to you, I definitely get irked whilst other people think about worries that they plainly don't recognize about. You managed to hit the nail upon the highest as neatly as outlined out the whole thing without having side effect , folks could take a signal. Will likely be back to get more. Thank you

my web page - private krankenversicherung beiträge
Here is my web page : private krankenversicherung beiträge

Anonymous,  January 4, 2013 at 3:56 PM  

Hello there, just became aware of your blog through Google, and found that it is really informative.
I am going to watch out for brussels. I'll be grateful if you continue this in future. A lot of people will be benefited from your writing. Cheers!
Feel free to visit my weblog :: in die private krankenversicherung wechseln

Anonymous,  January 26, 2013 at 8:29 AM  

You really make it seem so easy with your presentation but I find this matter to
be really something which I think I would never understand.
It seems too complicated and extremely broad for
me. I'm looking forward for your next post, I will try to get the hang of it!
Also visit my site :: microsoft adcenter

Anonymous,  January 30, 2013 at 12:46 PM  

Woah! I'm really enjoying the template/theme of this website. It's
simple, yet effective. A lot of times it's hard to get that "perfect balance" between user friendliness and visual appearance. I must say that you've done a superb job with
this. Also, the blog loads very quick for me on Opera.
Exceptional Blog!
Feel free to surf my website : private krankenversicherung stiftung warentest

Anonymous,  February 1, 2013 at 2:02 AM  

Very soon this web page will be famous among all blogging and site-building
users, due to it's fastidious articles or reviews
My web blog bad credit home equity loans

Anonymous,  February 1, 2013 at 2:09 AM  

Hi there everyone, it's my first visit at this web page, and article is actually fruitful designed for me, keep up posting such content.
Review my web site - web hosting provider

Anonymous,  February 1, 2013 at 2:17 AM  

Definitely believe that which you said. Your
favorite reason seemed to be on the internet the simplest thing to be aware
of. I say to you, I certainly get irked while people consider worries that they plainly don't know about. You managed to hit the nail upon the top as well as defined out the whole thing without having side effect , people can take a signal. Will probably be back to get more. Thanks
My weblog jack jones outlet online

Anonymous,  February 6, 2013 at 6:27 AM  

I go to see everyday a few web pages and websites to read articles or reviews,
but this blog presents quality based writing.

Look into my weblog :: online kredit ohne schufa
Here is my page :: krankenversicherung für beamtenanwärter

Anonymous,  February 21, 2013 at 6:13 AM  

Hi there! This article could not be written any better! Reading through this post
reminds me of my previous roommate! He constantly kept talking about this.
I'll send this post to him. Fairly certain he'll have a great read.
Thanks for sharing!

my web-site internet marketing company

Anonymous,  February 24, 2013 at 9:59 AM  

hello there and thank you for your info – I have definitely picked up something
new from right here. I did however expertise several technical issues using this website, as I experienced to reload
the site lots of times previous to I could get
it to load correctly. I had been wondering if your web hosting is OK?
Not that I am complaining, but slow loading instances
times will often affect your placement in google and could damage your quality score if ads and
marketing with Adwords. Anyway I am adding this RSS to
my e-mail and can look out for a lot more of your respective fascinating content.
Ensure that you update this again very soon.

Check out my site; kredit arbeitslos

Anonymous,  February 26, 2013 at 3:43 PM  

Superb post however I was wondering if you could write a litte more on this topic?
I'd be very grateful if you could elaborate a little bit more. Cheers!

my weblog ... handy für jedermann trotz schufa

Anonymous,  February 27, 2013 at 10:41 PM  

Way cool! Some extremely valid points! I appreciate you penning this post and also the rest of the site is also really
good.

my web-site :: private krankenversicherung für beamte

Anonymous,  March 10, 2013 at 11:00 AM  

Thank you for the good writeup. It in fact was a amusement account it.

Look advanced to far added agreeable from you! However, how could we communicate?


Also visit my web blog - Http://Www.Lutonengineering.Com/Articles/Article-46091.Htm
My page - legitimate work home jobs

Anonymous,  March 14, 2013 at 9:58 AM  

Excellent web site you have here.. It's difficult to find good quality writing like yours these days. I honestly appreciate individuals like you! Take care!!

Here is my web-site; mouse click the up coming website
my website > http://ipstube.Ips.k12.in.us/users/ReaganTvz

Anonymous,  March 30, 2013 at 10:14 PM  

I have been exploring for a little bit for any high quality articles or weblog posts on this kind of house .

Exploring in Yahoo I at last stumbled upon this website.
Studying this information So i'm satisfied to convey that I have an incredibly just right uncanny feeling I found out just what I needed. I such a lot no doubt will make certain to don?t omit this web site and give it a look regularly.

Feel free to surf to my site simply click the up coming document

Anonymous,  April 13, 2013 at 10:39 AM  

I'm not sure exactly why but this web site is loading incredibly slow for me. Is anyone else having this problem or is it a problem on my end? I'll check back later on
and see if the problem still exists.

Look into my web-site :: cheap seo services

Anonymous,  April 25, 2013 at 12:10 PM  

Hi, i believe that i noticed you visited my site thus i came to return the
favor?.I'm attempting to in finding issues to improve my site!I assume its adequate to use a few of your ideas!!

Have a look at my site - business ideas canada