Saturday, November 8, 2008

Game Testers and Game Testing


Game Tester
A game tester analyzes video games to document software defects as part of a quality control process in video game development. While popularized as a dream job for gamers, interactive entertainment software testing is a highly technical field requiring computing expertise, analytic competence, thick skin, and the ability to endure long hours.

Description
Modern video and computer games take from one to three years to develop (depending on scale). Testing begins late in the development process, sometimes from halfway to 75% into development (it starts so late because, until then, there is little to nothing to playtest). Testers get new builds from the developers on a schedule (daily/weekly) and each version must be uniquely identified in order to map errors to versions. They also test the durability of the game disc through a series of tests that can include how much damage the disc can take before becoming unresponsive, and how glitches can affect how the game runs.
Once the testers get a version, they begin playing the game. Testers must carefully note any errors they uncover. These may range from bugs to art glitches to logic errors and level bugs. Some bugs are easy to document ("Level 5 has a floor tile missing in the opening room"), but many are hard to describe and may take several paragraphs to describe so a developer can replicate or find the bug. On a large-scale game with numerous testers, a tester must first determine whether the bug has already been reported before they can log the bugs themselves. Once a bug has been reported as fixed, the tester has to go back and verify the fix works.
This type of "playing" is tedious and gruelling. Usually an unfinished game is not "fun" to play, especially over and over. A tester may play the same game — or even the same level in a game — over and over for eight hours or more at a time. If testing feature fixes, the tester may have to repeat a large number of sequences just to get to one spot in the game. Understandably, burn-out is common in this field and many use the position just as a means to get a different job in game development. For this reason, game testing is widely considered a "stepping stone" position. This type of job may be taken by college students as a way to audit the industry and determine if it is the type of environment in which they wish to work professionally.
In software development quality assurance, it is common practice to go back through a feature set and ensure that features that once worked still work near the end of development. This kind of aggressive quality assurance—called regression testing—is most difficult for games with a large feature set. If a new bug is discovered in a feature that used to work, once it is fixed, regression testing has to take place "again".
Game testing becomes grueling as deadlines loom. Most games go into what is called "crunch time" near deadlines; developers (programmers, artists, game designers and producers) work twelve to fourteen hours a day and the testers must be right there with them, testing late-added features and content. Often during this period staff from other departments may contribute to the testing effort to assist in handling the load.
All console manufacturers requires that the title submitted goes through a series of rigid standards established. Failure to respect the required standard prevents the game to be published on the market. Many video game companies separate technical requirement testing from functionality testing altogether.


Types of Game Testing
Localisation testing is the testing that needs to be done to make sure the game is the correct language and all spellings, grammar and punctuation is correct. This is just a small amount that the localisation tester has to do.
Beta testing means any testing that’s done on a game that’s in the "beta" stage of development. Often, though, it means the first version of a game that anyone outside the company sees: a ‘public beta’. Public betas are effective because thousands of eager fans will often find bugs that your small group of testers couldn’t. It’s also useful to distinguish beta testers from QA testers: Beta testers might test from home, or casually on a single project, while QA testing is generally a full time, trained job.
Regression testing is a stage of testing. Once a bug has been fixed by the coders, it’s returned to the QA department for regression testing. QA checks to see whether the bug is still there (regression) and then runs similar tests to see whether the fix broke something else. That second stage is often called halo testing; it involves testing all around a bug, looking for other bugs.
Compatibility testing is for the hardware mechanics. In order to get the list of system requirements for a game, you need to test it on a bunch of different machines, each with specific video cards, amounts of RAM, operating systems, processors, and so on. You can often start with a good guess (if a game was built for Flash, it has the requirements that Flash does) but further testing is still required. Compatibility testing is generally done toward the end of development, when the game is coming together, because so much of the compatibility depends on exactly how the programmers work their magic. This means that if resources are available, it’s often good to run at least two rounds of compatibility tests: one early in beta to highlight some problems with plenty of time to fix them, and one late in beta or during GMC (gold master candidate, when the game is ready to ship) to really figure out exactly what the system requirements are.
Load testing tests the limits of a system. The system could be the number of players on an MMO server, or the number of sprites active on the screen, the amount of traffic running over an internet connection, or the number of threads running in a particular program. It’s most commonly used in the first sense, to mean the number of players or processes running on a server. Load testing generally requires a lot of people (like a beta test group) or software that fakes heavy activity.
Unit testing is run on a piece of code, often automated, to test whether it works at a basic level. For instance, if a piece of code is supposed to take in a username for an MMO and return an error if the name is already taken, then the unit test would automatically input a name that is known not to be used and one that is used and make sure that the proper response is given. Often programmers write unit tests for their own code, and QA only gets to see that part of the game once it has passed all of its unit tests. A good QA department, though, may help design unit tests to make them rigorous. Unit testing happens throughout a project at anytime the programmers are working.
Soak testing involves running a game for a long time to see whether performance degrades. Very subtle bugs like memory leaks might not come out in playing the game for five minutes but could ruin a saved game or crash the computer after an hour and a half of playing. Soak testing is often automated, with software faking mouseclicks or otherwise making sure the computer doesn’t go to sleep. Many soak test bugs end up being filed unfixed as PUB ("Psychotic User Behavior") because a normal player wouldn’t play the same Pac-man level for 175 minutes in a row.

Responsibility
In the early days of computer and video games, the developer was in charge of all the testing. Since most games were so limited in scope, this was easy and usually required no more than one or two testers. In some cases, the programmers themselves could handle all of the testing.
As games have become more complex, a larger pool of quality assurance (QA) resources is necessary (sometimes it is called "Quality Assessment"). This being the case, most publishers have a large QA staff that they have testing various games from different developers.
Usually one group of testers will work on the same game from the beginning of the QA process to the time the game ships (goes "gold"). Thus, they become experts at the game and become familiar with all its nuances and weaknesses. Normally a group of testers will work on from one to two games at a time, depending on each game's scale. As one game nears completion, they may focus more time on it as the QA requirements escalate. Near the end of development, the QA staff may relocate to the development location in order to provide intensive QA work and be easily accessible to developers.
Despite the large QA infrastructure most publishers have, many developers will retain a small group of testers to provide on-the-spot QA from time to time.

Compensation
Despite the job's difficulty, game testing doesn't pay a great deal and is usually paid hourly (around USD$10 - $12 an hour). Testing management is usually more lucrative, but this type of job usually requires years of experience and some type of college degree. For this reason, as mentioned earlier, most game testing jobs are taken as "foot in the door" positions, used as a stepping stone for more lucrative lines of work in game development. An annual survey found that testers earn an average of $39,063 annually. Testers with less than three years experience earn an average of $25,142 while testers with over six years experience earn $43,056. Testing leads with over six years experience earn on an average of $70,658 a year.
Some employers offer bonuses for the number of bugs a tester finds. While this approach may seem initially reasonable, it often leads to testers reporting game features they personally don't care for as bugs, or reporting features as bugs simply for the extra money.

0 comments: