不要让自动化测试失控

I.M.Testy的一篇文章《UI Automation Out Of Control》对于不顾一切，企图自动化测试一切的人们而言应该读读：

http://blogs.msdn.com/imtesty/archive/2009/08/01/automation.aspx

 

作者最后归纳了几个自动化测试的要点（非技术要点）：

• Not all automated UI tests save time!
  Tests that require constant massaging and tweaking because they constantly throw false positives take up a huge amount of a tester’s time in wasted maintenance.

考虑到维护的时间，并非所有自动化测试都能节省时间。

• Sometimes a human is a more efficient oracle than a computer algorithm!
  Sure, just about anything a computer does can be automated to some degree in some fashion, but there really are clearly some tests where it is more prudent and simpler to rely on a tester.

有时候，用人来判断比用计算机算法更简单、更可靠！

• Don’t rely on automation to emulate your customers!
  Test automation does not effectively emulate a human user. Sure, we have test methods in some of our internal automation frameworks to slow down simulated keystrokes (the actual keys are not being pressed on the keyboard), or simulate multiple or repeated clicks on a control or the mouse, and other tricks that try to emulate various user behaviors; however, test automation is generally poor at detecting behavioral issues such as usability, ease of use, or other customer value type assessments. Rely on the feedback from internal and external customers who are dog-fooding, self-hosting, and beta-testing your product (and act on it).

不要依赖自动化来模拟用户行为！尤其是在做可用性、易用性测试时。

• Go under the covers!
  I think many testers rely too heavily on UI automation because they think it emulates user behavior (although most things such as populating a text box are simulated via Windows APIs), or perhaps because they don’t know how to dig into the product below the surface of the UI. Whatever the case, think about the specific purpose of the test. If it is easier to check a return value, or call an API to change a setting then go deep…and stop messing around on the surface. (It only complicates the test, wastes valuable machine cycles, reduces reuse across multiple versions, and often leads to long term maintenance costs. (For an example of this see my previous post.)

不要忘了我们还有单元测试！

• Constantly massaging code contributes to false negatives!
  I have seen many cases where a tester designs a a UI automated test, and then tweaks a bit here and there to get it to run. Often times this tweaking contributes to a tests ineffectiveness in exposing problems, and may even hide other problems. Also, some tweaks are geared around synchronization issues (sync’ing the automated test with the system under test) and involve artificially slowing down the automation (usually by stopping or ‘sleeping’ the automated test process for a specific period of time). Other tweaks might hard-code parameters that then make the test fail on a different resolution or non-portable across different environments.

UI自动化测试往往倾向于不断调整脚本以适应被测试对象，这些调整往往导致问题暴露不够高效，甚至隐藏了某些问题。

• STOP trying to automate every damn test!
  As I stated before…just because we can automate something doesn’t mean that we should try to automate everything! We need to make rational decisions about what tests to automate, and what is the best approach to automating that test.

切忌自动化所有测试！