Web Service Test Automation Standards

Tools
SoapUI Pro and/or open source (http://www.soapui.org/) for development/or RFS
Jenkins to run SoapUI/RF tests

Test Goal

• Cover all URL endpoints for Health Test and Bad Request
• Cover all HTTP statuses served by the service (positive and negative)
• Cover basic request response requirements.
• Cover incremental URL endpoints along with its variations for rest of test types.
• Cover data variations that results response variations.
• Cover use cases that simulates user behavior from an application.
• Cover performance test using various strategies

Test Types
1. Sanity Test
Health Test (Checks for service does not return HTTP status 500 and 503)
Bad Request Test (Checks for service handles negative requests with service defined appropriate HTTP status codes)
Service Sanity Test (Checks for required responses exists and not null, data types where appropriate, requests node data matches response node data if exists)
2. Functional Test (User scenarios on client side that can be achieved by series of service requests - Helpful for isolation test)
3. Regression Test (All Service Sanity and Functional tests)
4. Smoke Test (Health and bad request test types and used for post deployment)
5. Data Driven Test (Cover service sanity test with data variations)
6. Performance Test
- Simple Strategy
- Fixed Rate Strategy
- Variance Strategy

Target Environment To Run Test & Schedule
All Environments
Smoke test must run after every deployment
Development
All tests run on demand
QA
Regular scheduled test (All types)
Staging
All test run on demand
Production
Health, Bad Request and Sanity test types should run against production. This comprehensive check should be scheduled 3 or 4 times a day to run against each production server. Scheduled time must be agreed upon QA, product owners.
Test Data
Test data running in QA, Staging, and Production environment should be controlled data set. These controlled data set should be agreed by QA and Product owners.
Test data for Production should also be configurable so that when needed for quick test, data could be passed as a parameter.

Regression testing、Smoke Test、Sanity testing定义：
http://blog.sina.com.cn/s/blog_45237cb10100ivc5.html