Automation Body of Knowledge (ABOK)--测试自动化知识体系

 

测试自动化的概念愈演愈烈，但是往往大家容易把测试自动化理解为选取一些工具，然后把手动的case自动化的执行起来就叫做测试自动化。其实上面说的应该只能算作测试自动执行。测试自动化是一套完整的从产品评估、自动化范围选取、自动化策略定义、工具选取、case管理、结果分析等完整的流程。所以要真正的做好测试自动化，需要有完整的相关知识结构。ABOK可以说给大家一个相对清晰、简短的介绍，帮助大家有更清晰地了解。

 

The Automation Body of Knowledge is a tool-independent skill set designed to help software test automation professionals address automation challenges that are present in the world of software testing. （ABOK是独立于工具的技巧集，设计的目的是帮助软件测试自动化工程师处理现实世界中的软件测试自动化的挑战。）Automated Software Testing is a discipline that is separate from Manual Software Testing, and must be treated as such. The ABOK acknowledges this, and provides engineers with a way to assess, improve, and better market their test automation skills more effectively than tool-specific benchmarks can do alone. This body of knowledge may also be used by organizations as specific criteria for more effectively assessing resources and establishing career development tracks. Not every automation engineer is required to be proficient in every one of the ABOK skill categories, but knowledge of the different skill categories is essential in the continued improvement, growth and development. It is recommended that each test automation effort, however, have a team that represents most, if not all of these skills, whether the team is composed of one or many resources.

The Skill categories are listed below, and may be clicked on to get a summary of each category. Skill categories 1 through 7 (macroscopic) include skill concentrations for automated test leads, while skills 8 through 12 (microscopic) include skill concentrations for automated test engineers. Given the state of many software projects, the Test Lead depends on the Automation Engineer for answers regarding automation planning, therefore the Automation Engineer may also want to consider a concentration in skills 1 through 7 as well. 

 

Skill Category 1: Automation's Role In The STLC

Being successful in test automation first requires an understanding of what test automation is, and where it fits into the overall testing lifecycle。

（要明确自动化带来的好处，没有benefit的事情不要做^_^）

Difference between software testing and software test automation

 

Software Testing is a discipline in which test logic is designed to be implemented by a human being. As opposed to, Software Test Automation designs script logic that allows that manual test logic to be implemented by a tool. An expansion of this concept asserts that software test automation involves tool support for all aspects of a test project, not just automation of test execution. With that being said, as a test automator, you should be aware of all phases of the testing lifecycle, along with tools that support those phases. （测试自动化不仅仅是测试执行自动化）

Test Tool Acquisition and Integration

 

The Test Tool Acquisition is the process of identifying the tool goals and objectives, making an effective business case and narrowing down a group of candidate tools to the selection(s) that best meet(s) the organizational needs. Test Tool Integration is, conversely, the process of gradually addressing tool considerations and widening the tool’s acceptance, use and benefits.The various sources provide extensive directions on how to conduct the tool selection and acquisition process. Most sources will agree, however, that this process should at a minimum include the following:

1. Making a well-informed decision to automate

2. Conducting a test tool evaluation

3. Addressing tool implementation considerations

4. Piloting the tool implementation

Often, organizations are pushed into making a decision about the introduction or expansion of test automation based on the ability to buy a specific automated test tool. It’s important to expand your vision to be able to identify several options – both commercial and non-commercial – and to be knowledgeable on how to perform an effective evaluation of these options so that a more informed decision can be made about test automation implementation.  

Automation Benefits and Misconceptions

Behind every decision to introduce test automation is a wide variety of stated or implied goals and expectations for what the automation must accomplish. Without a proper understanding of reasonable test automation goals, the automation effort will not survive, and will be destined to sit on the proverbial shelf of your test organization. Understanding the impact of test automation means having a firm grasp on the benefits of test automation, accompanied by an understanding of the common misconceptions surrounding test automation. Being equipped with this knowledge helps a Test Automator more effectively implement a test automation effort, “sell” the automation to the powers-that-be by presenting and tracking a business case, and manage effort-ending misconceptions and unrealistic expectations held by the organization decision makers. 

Automation Return-on-Investment (ROI)

Software test automation is an investment in time and money, and like any investment, the stakeholders – normally managers or customers – are expecting a positive return from that investment. If their expectations are not sufficiently met, stakeholders will probably discontinue the investing in automation. The investment is justified by identifying the potential return-on-investment (ROI), a ratio of benefits to costs. There are several approaches for calculating ROI including:

·      Simple ROI Calculation – A percentage calculated in terms of the monetary savings that test automation provides.

·      Efficiency ROI Calculation – Examines test automation benefits in terms of time that is saved on certain portions of the testing effort.

·      Risk Reduction ROI Calculation – A percentage calculated in terms of the increased quality that results from the increased test coverage provided test automation.

Skill Category 2: Test Automation Types And Interfaces

Test Automation Types

For simplicity the different types of test automation – functional (regression), unit, integration, performance, etc. – are often lumped into one ‘test automation’ category. It’s important to understand some of the basic differences among these different types, even if you don’t specialize in all of them, so that you can speak intelligently about them when questioned by members of the organization. Since all automators are often “painted with the same brush”, being totally oblivious to basic concepts in each type of automation can negatively impact all automation efforts in the organization. In addition, being able to display minimal knowledge in various types of automation may provide management with enough confidence in you to allow you to pilot a new automation effort, which would help you to ultimately expand your skills. 

Test Automation Interfaces

The main interfaces made available for application automation are:

·      Command Line Interface

·      Application Programming Interface

·      Graphical User Interface

The Graphical User Interface is probably the most popular interface for automation implementation, due to the fact if more closely simulates how a user might use the tool. The Command Line Interface and Application Programming Interface, however, are advantages given the fact that they make it simpler to implement test automation.

Skill Category 3: Automation Tools

A test automation professional also must be prepared to provide insight into tools that support all aspects of the testing lifecycle. This requires an understanding of the different types and categories of tools that support the testing lifecycle, as well as the criteria for assessing these different types of tools.

These different types of tools include:

·      Software Configuration Management Tools

·      Business/System Modeling Tools

·      Requirements Management Tools

·      Unit Testing Tools

·      Test Management Tools

·      Defect Tracking Tools

·      Code Coverage Analyzer Tools

·      Functional System Test Automation Tools

·      Performance System Test Automation Tools

Skill Category 4: Test Automation Frameworks

Automation Scope

Total automation cost is composed of both development costs and maintenance costs.  As the automation framework becomes more defined, scripting increases in complexity. While this causes development costs to increase, it also causes maintenance costs to decrease. As the scope widens, maintenance becomes increasingly important. Also, as maintenance becomes increasingly important, the automation framework must be more advanced in order to reduce total automation costs. Therefore, before making a determination of the framework that will be used for test automation, an evaluation must be conducted on the implied and/or stated scope of the organization’s automation effort.

Roles and Responsibilities

Each framework type will have one or more of the following roles for creation and implementation of the framework:

·      Team Lead

·      Test Engineer

·      Lead Automation Architect

·      Cross Coverage Coordinator

·      Automation Engineer (Automator)

Very often, one person may hold multiple roles.

Framework Generations

Over the years, automated testing has evolved, becoming increasingly defined and sophisticated with each new evolutionary phase.  This evolution may be discussed in terms of three generations:

·      First Generation – This generation is what this book refers to as the Common Approach.  It is primarily driven by record and playback.

·      Second Generation – The second-generation learns from the problems of the first generation.  It involves creating a framework in which to automate tests. It also incorporates the concepts of functional decomposition, which simply means creating modular functions for executing fundamental actions that occur in the execution of the tests in a given test bed.  These actions may then be called upon and executed independent of one another.  This generation also more greatly separates test data from automation code, through greater use of parameterization.

·      Third Generation – The third-generation of automation builds upon the concepts of the second generation.  It strongly depends on functional decomposition, but evaluates the functions at a much higher level of abstraction, so that code may be reused by tests within the same application, and by tests in different applications.  This is accomplished by separating test data and specific application functionality from automation code.

During the design phase a determination must be made about which generation will define your automated test framework

Skill Category 5: Automation Framework Design

 

The process of designing an automated test framework is not an exact science. It is possible to definitively identify the different types of frameworks, but the process used to select and implement a particular framework is a little more difficult to pinpoint in such a way that most of the industry experts agree. The most important thing however, at this point in automation history is to ensure that a well thought out approach based on common, successfully implemented industry practices is used and honed within a given organization. This skill category identifies an approach, from a high enough level that it will fit with where the IT industry currently is relative to test automation, while still remaining low-level enough to be useful for implementation. This approach involves the following steps:

1.   Selecting a Framework Type

2.   Identifying Framework Components

3.   Identifying Framework Directory Structure

4.   Developing Implementation Standards

5.   Develop Automated Tests

Skill Category 6: Automated Test Script Concepts

Test Selection

Choosing what and when to automate involves an analysis of the following:

·      Automate items that are tedious for manual execution, but relatively easy to automate.

·      Items that need to be executed over and over again.

·      Items that will increase coverage beyond what will realistically be done manually

In addition, the decision depends on the goals of the organization such as cost goals, efficiency goals, and risk mitigation goals. 

Automated Test Design and Development

Automated test design and development are tied directly to the interface in which the tests will be created, the way quality attributes are applied at the test level, the elements that are common to automated tests, and the standards used for creation of tests. 

Automated Test Execution, Analysis and Reporting

The biggest items to address regarding test execution are:

·      How many machines will be used for automated test execution

·      Error handling

·      How the results are reported and analyzed

Skill Category 7: Quality Attribute Optimization

This category defines various quality attributes of the automated test suite and identifies ways of addressing these attributes based on priorities and constraints surrounding each.

The following are some Quality Attributes that may be considered.

·      Maintainability

·      Portability

·      Flexibility

·      Robustness

·      Scalability

·      Reliability

·      Usability

·      Performance

Skill Category 8: Programming Concepts

 

Whether you use a tool with a scripting language, tree structure and/or keywords, fundamental programming concepts remain paramount in effectively automating tests, and increasing the distance the test can cover through increased system coverage and test flexibility. Concepts such as variables, control flow statements (if..then..else, for..next, etc.), and modularity are discussed in this category.

Skill Category 9: Automation Objects

Recognizing Application Objects

Once upon a time, functional software test automation primarily used an “analog” approach that relied on specific coordinate locations of a screen or application window for performing mouse and keyboard operations that simulated user activities on the application. This was slightly unreliable, and difficult to maintain on a large scale, because when the screen, window or application components moved ever so slightly, the automated test would fail because it was trying to operate on an element in a position in which it no longer existed. Modern test automation conversely takes a different approach that locates the objects on the screen based on properties that define that object and then performs the desired operations once the object is found. 

Object Maps

One of the simplest ways to boost the maintainability and robustness of an automated test suite is through the introduction of Object Maps. Object Maps reduce the amount of information that is being maintained in an automated test by storing object properties in an external file and referencing objects according to variable names (also known as Logical Names) that are then tied to the properties. 

Object Models

Many software applications are composed of several interrelated objects, and an object model is an abstract representation of the hierarchical group of related objects that define an application and work together to complete a set of application functions. The advantages offered by object models include: 

·      Increased application understanding

·      Increased scripting power

Dynamic Object Behavior

One of the biggest challenges faced by automated test engineers is that of dynamic object behavior. People process information about the application based on visual inspection, while automated tests use object properties. People can, therefore, easily make adjustments when there is a slight change from a visual standpoint, and can ignore many property changes. The computer, however, cannot adjust as easily. The necessary adjustments have to be anticipated up front and programmed.

 

Skill Category 10: Debugging Techniques

Types of Errors

Regardless of how well automated tests are designed and created, problems will occur. Sometimes the problems are related to the scripts, and sometimes they are related to the application, but the root cause is not always simple to find. The inability to effectively debug scripts can severely delay schedules, and can even bring automation to a screeching halt.  The first step in script debugging is in understanding the types of errors that may be encountered. There are 4 main generic types of errors that may be encountered upon script execution:

·      Syntax Errors

·      Run-time Errors

·      Logic Errors

·      Application Errors

Debugging Techniques

The trickiest part of debugging is finding out the true source of an error. This involves ensuring the error is reproducible, and then the process of finding the main source of the error must begin. Very often what seems to be the source of the error is actually just a symptom of the true error, so there are several techniques that may be employed to find the source of an error. At that point a determination may be made on whether or not the error is due to an application failure or a script issue. Then solutions for fixing the error may be addressed. Effective debugging typically mirrors the following process: 

·      Identifying error existence

·      Reproducing the error

·      Localizing the error

·      Fixing the error

Skill Category 11: Error Handling

 

Error Handling Implementation

Error handling is implemented in a variety of ways within automated test scripts, most of which can be summarized in the following three categories:

·      Step Implementation

·      Component Implementation

·      Run Implementation

Error Handling Development

Error handling routines are critical for test automation, particularly for effectively resolving runtime errors and other unexpected system events, because excessive runtime errors are the enemy of test automation robustness. The figure below illustrates an error handling development process that may be used for implementing a successful error handling approach. These steps include:

·      Diagnose Potential Errors

·      Define Error Capture Mechanism

·      Create Error Log Data

·      Create Error Handler Routine

Skill Category 12: Automated Test Reporting

Test reporting and analysis is an extremely repetitive process, yet can be extremely time consuming. This category addresses what types of reports could be generated and how to effectively produce these reports, so that time may be saved in the analysis and reporting. The categories of reports that may be generated by an automated test framework include:

·      High-level (Suites/Tests) reports

·      Low-level (Verification Points) reports