Chapter 8 Configuration Testing

Configuration Testing
Chapter 8
© attachmate 2004
Highlights
 Why configuration testing is necessary
 Why configuration testing can be a huge job
 A basic approach to configuration testing
 How to find the hardware you need to test
with
 What to do if you’re not testing software for a
desktop computer
© attachmate 2004
An Overview of Configuration Testing
 Configuration testing is the process of checking the
operation of the software you’re testing with all these
various types of hardware
 Different configuration possibilities
 The PC
 Components
 Peripherals
 Interfaces
 Options and memory
 Device drivers
© attachmate 2004
An Overview of Configuration Testing
(2)
 Which of the configuration areas would be
most closely tied to the program
 Graphical computer program
 A greeting card program
 A fax or communication program
 Why this all necessary?
 Standards aren’t always followed
 Software doesn’t work correctly with certain
hardware configurations
© attachmate 2004
Isolating Configuration Bugs
 The sure way to tell if a bug is a configuration
problem
 Perform the exact operation on another computer with
a different hardware setup
 Where the configuration problem lies
 Software may have a bug that appears under a broad
class of configurations
 Software may have a bug specific only to one
particular configuration
 Hardware device or its device drivers may have a bug
that only this software reveals
 Hardware device or its device drivers may have a bug
that can be seen with lots of other software
© attachmate 2004
Sizing Up the Job
 Configuration testing can be a huge
undertaking
 You may need to check every possible make
and model combination
 Too many to consider
 Example
© attachmate 2004
Approaching the Task
 Equivalence partitioning
 Reduce the huge set of possible
configurations to the ones that matter the most
 What makes the effort a success or not
 The information you use to make the decision
 Learn as much as you can and bring in other
experienced testers or programmers
© attachmate 2004
Decide the Types of Hardware You’ll
Need
 Look closely at your software feature set to
make sure that you cover everything
 Draw a table to summarize what hardware
pieces you need to put together to make it
work
 Online registration example
© attachmate 2004
Decide What Hardware Brands, Models,
and, Device Drivers Are Available
 Create a list of hardware to test with
 Check out recent equipment reviews
 PC Magazine or Mac World
 Research to see if some of the devices are
clones of each other
 Falling under the same equivalence partition
 Decide what device drivers you’re going to
test with
 Three approaches
© attachmate 2004
Decide Which Hardware Features,
Models, and Options Are Possible
 Software may not need to support all the
features of hardware
 If the configuration has less requirements,
simply not test these features
 Printer example
© attachmate 2004
Pare Down the Identified Hardware
Configurations to a Manageable Set
 Put all configuration into a spread sheet with
columns for the manufacturer, model, driver
versions and options
 Figure 8.7 example
 May test only the most popular printers or
ones that are less than 5 years old
 No right formula
© attachmate 2004
Identify Your Software’s Unique
Feature That Work with the Hardware
Configurations
 Test only those features that are different
from each other
 Different equivalence partitions
 Word pad printing example
© attachmate 2004
Design the Test Cases to Run on Each
Configuration
 Select and set up the next test configuration
from the list
 Start the software
 Load in the file configtest.doc
 Confirm that the file is displayed correctly
 Print the document
 Confirm that there are no error messages and
that the printed document matches the
standard
 Log any discrepancies as a bug
© attachmate 2004
Execute the Tests on Each
Configuration
 It’s often difficult and time-consuming to
identify the specific source
 Closely work with programmers and white-box
testers
 If the bug is specific to the hardware
 Consult the manufacturer's website for
information on reporting problems to them
© attachmate 2004
Return the Tests Until the Results
Satisfy Your Team
 It’s not uncommon for configuration testing to
run the entire course of a project
 An increment testing process
 Configuration test complete
 Get to a point where there are no known bugs
or to where the bugs that still exist are in
uncommon or unlikely test configurations
© attachmate 2004
Obtaining the Hardware
 Buy only the configurations that you can or will use
most often
 Always have different configurations available to test
on
 Contact the hardware manufacturers and ask if they
will lend or even give you the hardware
 Send a memo or email to everyone in your company
asking what hardware they have in their office or
even home
 If you have budget, work with your project manager
to contact out your test work to a professional
configuration and compatibility test lab
© attachmate 2004
Identifying Hardware Standards
 Knowing some details of the hardware
specifications can help you make more
informed equivalence partition decisions
 For Apple hardware
 http://develper.apple.com/testing
 For PC
 http://www.microsoft.com/whdc/system/platform
 A set of standards to receive the Windows logo
 http://msdn.microsoft.com/certification
 http://www.microsoft.com/whdc/whql
© attachmate 2004
Configuration Testing Other Hardware
 Testing other special software
 What external hardware will operate with this
software
 What models and versions of that hardware
are available
 What features or opinions does that hardware
support
 Then follow the same technique mentioned
above