Lessons Learned in Session-Based Exploratory Testing
Contents:
- Quick Overview
- Freeware Tools on the Net for ET and SBTM
- SBTM Critical Success Factor – Senior Required for Test Lead
- SBTM Critical Success Factor – Debrief Every Day
- Writing Good Test Notes Takes Time and Practice
- ET.XLS Tips
- Testing History at Your Fingertips
- Mine Your Data – Create Test Guides
1. Quick Overview
What is SBTM? It is a way of managing your Testing effort that isdifferent from how you probably learned to do it. Here are some links toprovide you with some background information:
- The SBTM home page at Satisfice
- Jon Bach’s presentation on ET and SBTM
- James Lindsay’s “Adventures in Session-Based Testing”
- Wikipedia entry on Session-Based Testing
- Satisfice: More articles about Exploratory Testing
- Article: Applying Session-Based Testing to Medical Software
According to the Satisfice SBTM home page, Session-Based Test Managementis “a method for measuring and managing exploratory testing.” In a nutshell, itis a Test Management Framework. It is one of many that you can choose to useand apply in your particular situation. I happen to like it.
Note that some people refer to it as SBT (Session-Based Testing), and someinclude this as part of “xBTM” which also includes “Thread-Based TestManagement”. TBTM is an alternate way to manage Exploratory Testing (ET) thatdoesn’t focus on time-boxing the test sessions.
2. Freeware Tools on the Net forET and SBTM
This list was last updated in September 2011.
| Name | Technology | Notes |
| Perl, DOS Batch files | The original command-line scan tool. (No changes since 2000.) | |
| Ruby (installs on top of the original Session Scan Tool) | Add-on for the original Sessions framework. Scan Tools are in Ruby – includes bug fixes and enhancements. (last updated 2009) | |
| Ruby, batch files and shell scripts | Version 2.0 of the Ruby command-line “Sessions” framework. | |
| Windows .Net 3.5 | Exploratory note-taking tool. Very slick with some nice integrated features. | |
| Java | Next step up from the original SBTM framework. Nice integrated interface. I like the built-in Review/Debrief workflow. | |
| Java | Working prototype. Cool features. Looking for programmers to continue development. | |
| Web app: Apache, MySQL, Perl | (I haven’t tried it yet – no comments) | |
| Java, MySQL | It’s Swedish – I translated the page to read it but haven’t tried it yet. |
TestExplorer is another tool that warrants special mention but it isnot freeware. It is a Windows desktop application with a rich feature set. Youmay download it and “try before you buy” to see if it fits your needs.
3. SBTM Critical Success Factor –Senior Required for Test Lead
About six months after I started using SBTM at one company, I gave anoverview presentation to the Development Team on how we do our testing. Iwanted the developers and leads to be familiar with the new terminology that weused – for example, get used to hearing things like ‘sessions’ and ‘debriefs’instead of words like ‘test plans’ and ‘test cases’. Most importantly, I wantedto get across the importance of an ‘uninterrupted session’. Distraction levels were generally highbut after the presentation the situation improved as people started asking usif we were “in a session” before interrupting us to do or discuss somethingelse.
Sometime afterwards, I talked with the VP of R&D to get some feedbackabout our testing approach. He was generally very pleased and said somethingthat I hadn’t thought much about until that moment. He said that the success ofour approach definitely depended upon having a good Senior person in the TestLead role managing the testing effort.
Hmm. This made me think. It just so happened that I had many yearsexperience as a Team Lead, and I was definitely Senior in terms of my testingskills and knowledge, so was it a coincidence that we were successful? Or werewe successful because I had both the passion and experience required tosucceed?
What are some of the things I do in the Test Lead role in Session-BasedTest Management?
- I’m persistent and meticulous when debriefing the reports. The scan utility can scan the session sheets for missing or unexpected elements, but it can’t tell you that a tester has forgotten (for the umpteenth time) to select the correct Areas for their session. How do you handle that?
- For that matter, when I look at a session report, I always view it with an audit/review in mind.Would this session report provide enough information to stand up to an independent audit at some point in the future when we’ve moved onto some other project and no longer remember what we did on that particular day?
- Constant vigilance! (Note that it may not always be your requirement for high accountability, but if it is then this point is relevant.)
- Bridge the terminology gap between the Testing effort (estimation and progress) and the Project Manager (PM).We use a Risk-Based Test Strategy for a given release that generally slices the work in a different way from how the PM’s Work Breakdown Structure (WBS) describes it. How do you bridge that gap to ensure that everyone knows where we are? This is about more than just a Traceability Matrix, but yes,the Test Lead is the living Traceability Matrix between the tests performed and the Release Requirements. We also estimate effort in terms of ‘sessions’ while the PM talks in terms of ‘hours’ and ‘days’. Some math may be required.
- Help decide when ‘enough testing’ has been performed on an area and move onto a new area. A lot of factors and experience goes into helping you develop a ‘good feeling’ about when it’s time to move on and/or decide that a particular charter has been adequately covered to sufficiently reduce the risk you started with.
- The ‘risk’ I speak of here is the risk of ‘not knowing’. We test a particular area because we feel there are certain risks about it. As we test we learn about the features and the system, report any bugs or issues, and become confident in our coverage of that area.
- When you feel that the ROI of your time and resources will be better spent exploring a different area of the application or system, it’s time to move on. How do you make this decision?
- Mentor and Coach testers on test techniques, tools, and other ideas to help them improve the efficiency by which they can get their job done. I love this part. This is where I have the most fun and see the biggest improvements in the testing effort as a result.
- Review, audit and maintain the data to support the SBTM metrics. If you don’t stay on top of these, you can’t put much faith in the numbers produced. I like to regularly review these metrics throughout a development project to give me a general feeling of where we are andhow we’re doing. I only share these metrics with my team members. No one sees them outside of the team. They’re for our use only, and that’s something else I do – protect the team by not leaking any metrics that might be misused against us.
There are more things that I could probably include, but the list above isa reasonable start. Is there anything there that I wouldn’t expect of a Test orTeam Lead managing their testing effort using a different approach? No, notreally. But that’s the point isn’t it? The Tools are a bit different and thesteps or processes are not spelled out for you. Thinking is required.
I think it’s a valid requirement to say that someone in the Test Leadposition needs to have some prior experience managing testing efforts. Theyshould have a pretty good idea of how to adapt to a new way of working, how toadequately support the team as they work, and how to communicate with managersoutside of the test team.
4. SBTM Critical Success Factor –Debrief Every Day
Don’t let these slip! When we first started with SBTM, there weretimes when we were extremely busy and putting in overtime hours that we wouldget behind on our session debriefs. It was easy to say that we could makebetter use of our time by doing testing rather than reviewing the testingalready done. That makes sense, right?
Well actually, no, it doesn’t work that way.
Here’s the thing: sessiondebriefs are like Code Reviews for programmers. Programmers don’t have to dothem either, but when you stop doing reviews you often notice the quality ofwork start to decline. When we finally got around to reviewing thebacklog of session sheets, we would often discover additional tests and risksthat would have been worth exploring. You don’t really want to get these kindsof epiphanies after a release has gone out the door. It’s too late then.
The ‘debrief’ aspect of SBTM is morethan just a quality check. It’s acomplement to your Exploratory Testing effort by putting your respective headstogether and revisiting the test strategy described in the Test Notes forelements of coverage, completeness, risk and repeatability. “Two heads arebetter than one” is the force at work here to increase your individual testingpowers. Each team member benefits as the testing notes are shared and theknowledge helps build newer and better tests as you go along.
We never skip these debriefs anymore. Even when the going gets tough anddeadlines are tight, we always make time every day to stop and debrief thesessions from the previous day. Personally, I like to do these first thing inthe morning. It gives me a good idea of the problems we encountered from theprevious day and lets me set or adjust the plan for the current day’s work.
There was a question I remember asking people during the time when I was aQuality Auditor years ago: “How do you know what you’re working on when youcome in on any given workday?” It was a question to get someone talking abouthow they set priorities, where they get their tasks from, lines ofcommunication, and so on. By doing the session debriefs in the morning, I knowthat I only have to worry about my session sheets being complete at the end ofany given day. If I have to stay late, I stay late and finish up what I needto. But when I go home I don’t generally worry about my work priorities for thenext day.I know that assoon as I come in, my session sheets will be reviewed and I will review otherpeople’s sessions, and within the hour we will all know what we’re working onfor the rest of the day. Risks discussed, test strategies confirmed oradjusted, priorities clarified, we’ve got the game plan… it’s time toRock and Roll!
Don’t let these slip. Do the session debriefs every day.
5. Writing Good Test Notes TakesTime and Practice
The session sheet template has many important elements, none of which Iwould want to give up completely managing or tracking in some way or another.The single most important element for me has to be the ‘Test Notes’ sectionthough.
When I review a session sheet, I know that I can ‘approve’ it when I cansay to myself that I know more or less exactly what someone did for the‘Duration’ listed in the sheet. Whether it is one hour or four, if I can’t sayto myself that I know what this person has spent that whole time doing, thenthe session report is incomplete.
The analogy that I often give at times like these is that a session sheetis like a Science Report that you probably had to do at some point inelementary or high school. A typical Science Report has elements like thefollowing: Objective/Purpose, Materials and Methods, Data and Observations,Conclusions or Inferences.
A good session report has all the same elements, although they may be laidout in a slightly different way.
The ‘Charter’ is the ‘Objective’. Itsets thescope for the session and should help you know when testing is complete andit’s time to move on to something else. A Charter can be as specific or generalas you like to help you get the information you need in the time allotted.Testing is for a specific purpose – what’s that purpose?
The ‘Materials and Methods’, ‘Data and Observations’, ‘Conclusions orInferences’ all seem to fall into the “Test Notes” section. That makes thissection pretty important. So how do you get good at writing Test Notes to giveyou all this information? The same way you get to Carnegie Hall – practice,practice, practice!
A good note-taker is like a good Sports Commentator with a twist. A SportsCommentator gives you the play-by-play of the particular game or sporting eventthat you are watching. A really good commentator alsocommentates. Thatis, provide you with background information on the players, the team, or someopinions about how or why certain things might have happened.
A good Test Note-taker does the same thing, and also needs to provide sufficient information to maketheir test session repeatable. Sounds simple enough, right? Ha! Try it.
You need to start by being clear about your ‘Materials and Methods’(a.k.a. “The Setup”).
- Materials:（qf:准备数据&测试环境）
- What data did you use? Where did you get it from, generate it, transform it or store it?
- How is the system or application configured? What operating system, browser or hardware are you using? Any particular tools or automated scripts? And so on.
- Methods:（qf:测试步骤&测试方法技术）
- What steps did you follow? What areas did you explore? What inputs did you use? What Test Techniques did you apply? Why? How are you keeping track of all of the possible test factors influencing the testing of a particular feature? (i.e. using a list or test matrix, etc.)
- Are there multiple scenarios that you need to cover to complete a particular charter? How are you keeping track of these? (e.g. Test Coverage Outlines) Were there any particular tools that you used to gather information during this session? And so on.
Next there are the ‘Data and Observations’.
- Data:
- How are you capturing the information and data that you observe? Are you doing it manually or using a tool? If you are doing it manually, can you trust the consistency and repeatability of your measurements?
- If you are using a tool, are there any known bugs, problems or calibration issues that might produce faulty data? Do you even know how to use the tool properly to get the data you want? (Training may be an issue sometimes. Session debriefs should give you insights into these opportunities. Tool ‘calibration’ may also be an important consideration.)
- Can you chart the data to develop a new understanding or insight into what the system might be doing? Are you familiar with analysis tools and charts to help you work with the data?
- Observations:（qf:测试依据来源&测试依据）
- How do you know that what you are observing is correct? If you start a test with certain expectations or assumptions, what are those expectations based upon? Are they based upon some reference like a specification or requirement? Is that information current and up-to-date?
- Are your expectations based upon a conversation you had with someone about how they think a particular feature should work? Does the application match those expectations? Even if they match, does that mean what you see is correct? How can you avoid Type I and Type II errors?
- Are there any biases or influencing factors in your observations that lead you to a certain ‘expected result’? Do you know how you might be able to identify these or work around them?
- Did you really see what you thought you saw? Did you get a screen capture? Can a log file or database audit-type table confirm certain events happened in the order you thought they happened? Where is this information stored so that it can be called upon during a later review if required?
- Was an observed result repeatable? Can you make it repeatable? If not, it’s still worth mentioning. Highlight the fact that it wasn’t repeatable so that you might share your observations with others.(qf:如果问题不可重现，需注明。)
Finally there are the ‘Conclusions and Inferences’.
- Conclusions:（qf:实际结果）
- Summarise what you have seen. There will always be the moments when you need to quickly skim a report to see if it contains the information you are looking for.Having a brief Test Summary or Synopsis either at the top or bottom of the test notes helps focus the reader to the important highlights required to complete that session’s charter.
- I’ve had teachers who have said that “nothing is ever conclusive, so don’t use conclusions – useinferences instead” and teachers who swear by “conclusions”. I think it’s a matter of timing really. I think you can draw conclusions based on the information you have at a given time to help you make a particular decision. How much you know about something changes over time, but at any given time you should be pretty certain about the information you have and what it means right now. I like to see comments like these scattered throughout the Test Notes. It tells me that the Tester is thinking about what they’re seeing and what it means at any given moment.
- A good tester can make their test notes better to read than a good book. I look forward to reading these good session notes and following along on their journey of discovery and exploration. That’s really cool.
- Inferences:（qf:建议）
- How do you feel about the session? If you have nothing ‘conclusive’ to say, is there something you caninfer?
- Some opinion is always better than no opinion. At the end of the day, Test Leads, Managers, and staff in other departments don’t have the time to rethink everything that you’ve already gone through. (If we did, we wouldn’t need you then would we?) Tell us what you think it means to you in plain & simple language, how you feel about it, and what recommendations you might make based on the information you have so far. Nowthat’s a good way to end a session report!
The above elements may seem like a lot to put into your Test Notes. To thenovice tester, it probably is. There’s a learning curve required as novicesstart off by putting too much information into the Test Notes section. To themore experienced tester, there are many shortcuts and assumptions that can beclarified in the notes to help you get to and focus on the important stuff. Forexample, I’ve read many good session reports that addressed all the elementsand questions above and were only a few sentences long.
How do you know what’s worth writing and what’s worth skipping orimplying? You need practice and a good Test Lead who cares about the qualityand repeatability of the session notes you produce. It isn’t easy, but it’s agood habit to develop. Remember that Session-Based Test Management is alsoreferred to as “High Accountability Exploratory Testing”. So if you’re notprepared to provide the ‘accountability’ then you might want to think aboutchanging teams.
In the end, when you get good at writing test notes, I think you’ll have alot in common with those good scientists who keep journals as records of theexperiments they perform and the discoveries they make. This isappliedCreative Writing. Put your ‘Thinking Cap” on because not only do you have tothinkto do your job, but you need to be your own personal commentator as you goalong and do it!
6. ET.XLS Tips
The original ‘sessions.exe‘ archive includes a samplespreadsheet called ET.XLS that can be used to generatesome interesting metrics based on the scanned session reports. It took me a fewprojects over the course of several months to figure out all the little detailsof how this spreadsheet works. The spreadsheet that I use now generatesconsiderably more charts and information based upon the submitted reports thanthe original/default spreadsheet. I highly recommend that you play with thisspreadsheet, get to know what the numbers and charts mean, and customise it toyour own needs.
Getting Started with ToDos
To begin with I tried to understand what each of the sheets/tabs in the ExcelWorkbook did so that I could understand what data would be useful to me in mycurrent context/situation. Not all of the details are documented and one itemin particular took me a while to figure out was the “TODO” sheet titled “TODOItems in the Hopper”.
I didn’t know how to get data into this report. The sequence of stepsseemed a bit odd, so here’s what I found out:
- Use the supplied ‘todo.xls‘ to come up with the list of topics and areas that you feel are important to cover during your testing project.
- Export or save the data to the ‘todos.txt‘ file. (My Ruby script above skips this step and just reads the Excel spreadsheet directly.)
- Use the ‘todo-maker.bat‘ file to generate empty session sheets that can be used as starting points for your charter sessions.
- The batch file creates these ‘et-todo-…ses‘ files in ‘c:\sessions\todos‘ folder by default
- When we use one of these ‘todo’ template files, we move them into the ‘c:\sessions\submitted‘ folder and work on them until they are complete and ready to be reviewed and eventually moved into the ‘approved‘ folder.
The ET.XLS file just never seems to detect any of the files in the ‘c:\sessions\todos‘ folder/hopper, so what gives?
It turns out that while step 3 puts the empty session reports into the ‘todos‘ folder, if you want these files to show up in the ‘hopper’ spreadsheetpage,you need to move them into the ‘approved‘ folder. Thenwhen you run the ‘scan-approved-then-run-report.bat‘ tool, itwill detect the ToDo sheets and generate the required report that can beimported into the ET.XLS spreadsheet.
Once you know that you can make adjustments accordingly. In the end, Idon’t use this spreadsheet to manage the ToDo hopper, I use another approach.It just took me a while to figure this tidbit out so I thought I would share ithere.
Charts – Just the Beginning
The nice thing about having these numbers in an Excel spreadsheet is that youcan then generate all sorts of charts and metrics to help you manage thetesting effort.
The “Exploratory Testing: Test Sessions” chart on the first “Summary”worksheet/tab took me a while to really grasp. Once I figured out that it isessentially a productivity chart that shows you the cumulative number ofsessions performed over time, I was able to use that information in ameaningful way.
A new chart that I include in my spreadsheet is the cumulative number ofBUGS reported (according to the submitted session reports) over time. I thensuperimpose that data series on top of the Summary sheet chart so that I cansee both thenumber of sessions and the cumulative number of bugsreported over time on the same chart. I now find this chart to be even moreuseful to me on the day-to-day management of our test projects.
I have other charts for coverage, but I keep those charts separate. Ithink I have more work to do when it comes to chart and metric analysis. Thegood thing is that I have the data and numbers to work with. I just need thetime to sit down and sift through it.
7. Testing History at YourFingertips
The session reports are like gold. We keep every session report we’ve evercreated both on the network and stored safely away in a document controlsystem. We haveinstant access to the testing notes going back tothe initial release of our flagship software from several years ago. That’samazing!
The first thing that really blew me away about SBTM was that allmy test records are stored electronically as simple text files on the network.I’ve been doing QA and testing software since the early 90′s and this is thefirst time that my test results have ever, really, truly been completelypaperless. Being something of an environmentalist that makes me somewhatpleased. The sheer efficiency of it is also marvelous for anyone who has everdone process improvement work before.
Talking about it is one thing, but I think an example is in order.
There was this one time when a Lead developer came up to me to talk abouta particular feature in the application that we were testing. He asked if Iknew how or when I tested it last because he couldn’t find any bug reports inthe system relating to that feature. I used my ruby ‘search’ script to scanthrough all the archived session reports on the network. (BTW, acolleague of mine uses a visual grep utility to do the same thing – sameresult, different tool.) Within a minute, I had called up several sessionreports from a year before (3 releases back) that described the full extent ofwhat we had tested, what we didn’t, and more importantly what bugs we hadreported during those sessions. With that knowledge we were able to devise anappropriate test strategy to complete the testing of the updated feature in thecurrent release we were working on.
It was surreal. Simply fantastic! I don’t ever recall having had theability to search through my complete testing records for the last 3 years soquickly to bring up the exact moment and details of when I had reportedspecific bugs down to the minute. I challenge anyone using more ‘traditional’test documentation approaches to live up to that kind of standard.
This is the information age, and “time is money.” If you can’t tell mewhat you’ve tested at any given moment over the last 3-5 years, how you testedit, who worked on it, what bugs you reported and any issues or problems thatyou encountered along the way, within minutes, then you need to take aserious look at the usefulness of your test management approach.
Text files aren’t glorious. They aren’t even particularly fancy. They areeasily readable on every major operating system that I’ve worked with, though,they also compress really well in archive (zip) files, and they are easilysearchable with a myriad of tools readily available on the internet. As aresult, I can find out the Who, What, Where, When, How and Why of any testingactivity our test team has ever performed since we first started keeping recordsseveral years and over a dozen releases back – all within Google time. That’sprogress!
8. Mine Your Data – Create TestGuides
This isn’t a part of Session-Based Test Management, but is such animportantcomplement to it that I feel I need to mention it here.
All these session reports you keep add up over time. So what do you dowhen the next major release comes around and you find yourself having toperform regression testing on the major features from the last release?
Well for a start, I wouldn’t recommend that you start exploring thosefeatures from scratch all over again. You already went through that exerciseonce in great detail and it probably took you anywhere from several days toseveral weeks to cover a particularly interesting or complex feature. Rememberthat SBTM is the framework to support your Exploratory Testing effort, and thatET is the simultaneousLearning, Test Design, and TestExecution of your application.
You already learned something when you tested a feature the lasttime, the first time you saw it. Testing the same feature over again will havea greatly diminished learning curve this time around. So how can youefficiently regression test a feature that took several weeks to cover the firsttime, in a timely manner this time?
Start by asking yourself what did you actually learn fromyour last testing experience? You need to compile the session reports thatcover a particular feature or area of the application in order to see thecomplete picture. When you bring them all together, you can identify the factsthat you need to develop a good test strategy starting point. You should beable to extract things like:
- How many testers did it take? How many sessions over how many days?
- Were there any particular system or environment configurations that you needed to test this feature or area properly? Any specific test data files?
- Additional equipment or hardware? Specific tools, scripts or software?
- What specific Risks were identified or did the Charters try to cover?
- What test techniques were used? Which ones were effective at finding bugs? Which were not?
- Did you have any checklists or tables to help you cover all the required elements or test factors affecting this feature? Do you haveany reusable Test Notes?
- Were there any tests that you skipped last time because you ran out of time?
- What bugs did you discover and report? What kinds of bugs were they? (i.e. simple UI spelling errors, complex mathematical/computational bugs, security or performance bugs, etc.)
- Were there any issues or roadblocks that appeared while you tested this area that could possibly arise again? i.e. did focussing on the particular area put an excessive load or stress on the system that prevented you from continuing to test for any length of time?
Once you have these questions and answers in mind, it’s time to bring themall together into a new document – a document I call a “Test Guide”. Thisdocument summarizes the best part of the testing strategy employed and what youlearned the last time you tested a particular feature or area of theapplication. We use MS Word documents for our Test Guides, we keep them simple& useful and anywhere from two to ten pages in length. Don’t use coverpages or Table of Contents or any of that fluff.
I should be clear that a Test Guide is just that – a guide – and containsno specific test cases. We sometimes describe interesting scenarios, but notest cases. I think I need to mention it again – NO TEST CASES! Individual testcases are useless unless used to illustrate an example of an application of aparticular test technique.
Could a Test Guide contain Use Cases? Maybe. Are these use casesdocumented anywhere else? If so, then go and hit yourself over the head withyour keyboard for asking such a question. A Test Guide wouldn’t be easilymaintainable if you duplicated information that is better maintained elsewhere.Just identify the link or location of where to get the other document(s) andlet someone else worry about maintaining the use cases separately from yourTest Strategy documents. If the use cases that youdiscovered duringyour testing are not documented anywhere else, I would recommendyou keep them documented separately anyway. If you have nowhere better to storethem, put them at the end of the document in an “Appendix” and just refer tothem in the right place(s) in your strategy guide.
When is a good time to work on creating these Test Guides? Good question.It depends. If you have to test a feature right away and you haven’t alreadygot a guide, then just review the past session reports as a prelude to completethe testing you need to do now. Perhaps you can create a Test Guide draft whileyou go through the material the second time around.
Generally, we work on these Test Guides in that down-time period after arelease has shipped and before the next development project is ready fortesting. Some people might call thistest planning. I would probablymore accurately call it “reviewing what we have learned to do it better andmore efficiently the next time.”
Now when we test an area that has been tested before, we just refer to thespecific Test Guide in our session report Test Notes section. It’s clear andefficient and helps us focus on what’s important: learning something new and notrelearning what we have already forgotten.
扫一扫
3785
被折叠的 条评论
为什么被折叠?
到【灌水乐园】发言
