12th WORKSHOP ON TEACHING SOFTWARE TESTING (WTST 2013)
JANUARY 25-27, 2013
MELBOURNE, FLORIDA
at the HARRIS INSTITUTE FOR ASSURED INFORMATION

The Workshop on Teaching Software Testing is concerned with the practical aspects of teaching university-caliber software testing courses to academic or commercial students.

WTST 2013 is focused on high volume automated testing (HiVAT). Our goal is to bring together instructors who have experience teaching high-volume techniques or who have given serious consideration to how to teach these techniques. We also welcome participants focused on the teaching of complex cognitive concepts and the transfer of what was learned to industrial practice.

As at all WTST workshops, we reserve some seats for senior students who are strongly interested in teaching and for faculty who are starting their careers in this area or beginning a research program connected with teaching this type of material.

There is no fee to attend this meeting. You pay for your seat through the value of your participation. Participation in the workshop is by invitation based on a proposal. We expect to accept 15 participants with an absolute upper bound of 25.

BACKGROUND ON THE WORKSHOP TOPIC

High volume automated testing involves automated generation, execution and evaluation of the results of a large set of tests. This contrasts with more traditional “automated” testing that involves automated execution of a relatively small number of human-created tests.

Here are four examples of the types of problems that underlie the need for HiVAT:

1. Many types of code weakness (such as timing-related problems) yield intermittent failures and are hard to detect with traditional testing techniques
2. There are immense numbers of possible combination tests of several variables together. Some combinations are unique (a failure appears only on a particular combination)
3. Some failures occur primarily when a system under test is under load and so detection and characterization is essentially a statistical challenge.
4. Characterizing the reliability of software requires a statistically useful set of tests.

In the academic community, the most commonly discussed HiVAT family of techniques is called “fuzzing” – but fuzzing as we know it involves very simplistic evaluation of the test results—essentially run the software until it crashes or fails in some other very obvious way. Other HiVAT techniques rely on more powerful oracles and can therefore find other kinds of bugs.

WTST is about teaching testing, not creating new techniques. The challenge we are trying to address in this WTST is that many of these techniques are known but not widely applied. We believe this is because they are poorly taught. As far as we can tell, most testing courses don’t even mention these techniques. Of those that do (and go beyond fuzzing), our impression is that students come out baffled about how to actually DO that type of testing in their work.

At Florida Tech, we’re trying to address this by creating “reference implementations” for several techniques—open source demonstrations of them, with commentary on the design and implementation. We’re hoping that WTST will provide examples of other good approaches.

The hosts of the meeting are:

• Cem Kaner (http://www.kaner.com and http://www.testingeducation.org)
• Rebecca Fiedler (http://bbst.info)
• Michael Kelly (http://www.developertown.com/)

Confirmed attendees include:

Name	Affiliation	Abstract & Presentation
Kaner, Cem	Florida Tech	An Overview of High Volume Automated Testing This overview describes the general concept of high volume automated testing (HiVAT) and twelve examples of HiVAT techniques. The common thread of these techniques is automated generation, execution and evaluation of arbitrarily many tests. The individual tests are often weak, but taken together, they can expose problems that individually-crafted tests will miss. The simplest techniques offer high coverage: they can reach a program’s weakness in handling specific special cases. Other techniques take the program through long sequences of tests and can expose programs that build gradually (e.g. memory leaks, stack corruption or memory corruption) or that involve unexpected timing of events. In my experience, these sequences have exposed serious problems in embedded software, especially in systems that involved interaction among a few processors. As we embed more software into more consumer systems, including systems that pose life-critical risks, I believe these tests will become increasingly important. The Insapience of Anti-Automationism
Karena, Catherine
Kelly, Michael	DeveloperTown	Workshop Facilitator
Fiedler, Rebecca	Kaner, Fiedler & Associates
Balasooriyi, Janaka	Arizona State University
Bedran, Thomas	Progressive Insurance
Demott, Jared
Gallagher, Keith	Florida Tech
Hoffman, Doug	Software Quality Methods	Results Oriented Teaching of High Volume Automation High volume test automation entails massive stimulation of the software under test (SUT). The mechanisms for such stimulation are straightforward once a tester becomes aware of the value of the approach and the techniques for implementation. Context matters, so the most applicable approaches and techniques are different for each situation. When I teach test automation I include high volume testing as one important approach. I present approaches and techniques using some experience examples. My emphasis about automated testing is on the oracles. This is especially important for high volume testing because of the huge variation in values and combinations. It may be easy to stimulate the SUT, but the test doesn’t provide much useful information if we can’t tell if the SUT is behaving reasonably. Therefore, I emphasize test oracles for high volume automation. (The techniques are applicable to all testing – human as well as automated.) The presentation will briefly introduce the high volume techniques and oracles, with an emphasis on the oracles that make the tests useful.
Hoffman, Dan	University of Victoria	Key Tradeoffs in High Volume Test Automation High volume test automation entails massive stimulation of the software under test (SUT). The mechanisms for such stimulation are straightforward once a tester becomes aware of the value of the approach and the techniques for implementation. Context matters, so the most applicable approaches and techniques are different for each situation. When I teach test automation I include high volume testing as one important approach. I present approaches and techniques using some experience examples. My emphasis about automated testing is on the oracles. This is especially important for high volume testing because of the huge variation in values and combinations. It may be easy to stimulate the SUT, but the test doesn’t provide much useful information if we can’t tell if the SUT is behaving reasonably. Therefore, I emphasize test oracles for high volume automation. (The techniques are applicable to all testing – human as well as automated.) The presentation will briefly introduce the high volume techniques and oracles, with an emphasis on the oracles that make the tests useful.
Robinson, Harry	Microsoft	Experience report from Bing High-volume automation shows great promise, but it baffles and frustrates traditional notions of testing and automation. For HiVAT to succeed, it has to come to terms with how people and organization think about and measure testing. If HiVAT can’t bridge that gulf, it will remain out of the industry mainstream. And lack of acceptance means that it will be hard for HiVAT practitioners to advance their own understanding and effectiveness. This presentation looks at a recent example of high-volume test automation in Bing. The effort was straightforward and found many interesting issues; the hardest part came in measuring the benefit, extending the approach, and making it useful to the wider community.
Sabourin, Rob	McGill University	Implementing Multi-thread Testing in Fitnesse
Tinkham, Andy	Magenic Technologies
Vaniotis, Thomas	LiquidNet
Xie, Tao	North Carolina State University	Teaching High Volume Automated Testing via Interactive Gaming
Doran, Casey	Florida Tech student	A Survey of the State of Testing in the Open Source Community This talk will present the results of a survey of many top open source projects. This study was conducted with an eye towards identifying real-world software for use as Software Under Test (SUT) in both lab environments (developing reference implementations of HiVAT testing tools), and classroom environments (lecture demonstration and homework or examination assignments in a testing or software engineering course). To this end, a grading system was developed around such features as a) the quality and testability of a project’s code, b) the availability of existing unit tests, c) whether the project uses regression tests as part of an automated acceptance procedure, and d) whether the project exposes an automation API, which could be useful in “long sequence”-style HiVAT. A complete list of surveyed OSS projects, including researcher notes, important project URLs, and other collected data will be provided to WTST participants.
Fiorvanti, Mark	Florida Tech student	High Volume Automated Testing in Security Testing HiVAT offers potential advantages to the security field as it allows security testers to identify and locate potential security flaws quicker and more accurately than standard techniques. The security testing community needs tools which will allow them to counter the asymmetrical nature of security, in which the adversary potentially only needs to exploit a single flaw to achieve their goal but the defender must mitigate all vulnerabilities to achieve their goal. Three main testing areas exist within the field of security: compliance/certification testing, penetration testing/ethical hacking and vulnerability research. HiVAT techniques are used within the penetration testing/ethical hacking and vulnerability research areas. Compliance/certification testing is not able to leverage these techniques as the information objectives are different than the other two areas of testing. Even though this technique is used, it may be subject to the misconception that HiVAT is simply just repeatedly running a large number of small tests. HiVAT can be effectively leveraged for vulnerability identification, countermeasure/filter evasion, and lastly exploitation resulting in security tests which are more thorough than common assessment techniques.
Frystacky, Michal	Florida Tech student
Fuller, Scott	Florida Tech student
Kabbani, Nawwar	Florida Tech student
Oliver, Carol	Florida Tech student	Reference implementations for high volume automated testing The Center for Software Testing Education and Research (CSTER) is focusing on building reference implementations of High-Volume Automated Testing (HiVAT) techniques in order to facilitate adoption of these techniques by testers who have found these approaches too awkward to implement from just a description. This talk will explore the details that these people will require of our reference implementations, as well as optional details that would greatly benefit their efforts to apply these techniques in their own environments.
Tereshchenko, Vadym	McGill University student	Implementing Multi-thread Testing in Fitnesse

HOW THE MEETING WILL WORK

WTST is a workshop, not a typical conference. It is a peer conference in the tradition of The Los Altos Workshops on Software Testing (http://lawst.com). Our presentations serve to drive discussion. The target readers of workshop papers are the other participants, not archival readers. We are glad to start from already-published papers, if they are presented by the author and they would serve as a strong focus for valuable discussion.

In a typical presentation, the presenter speaks 10 to 90 minutes, followed by discussion. There is no fixed time for discussion. Past sessions’ discussions have run from 1 minute to 4 hours. During the discussion, a participant might ask the presenter simple or detailed questions, describe consistent or contrary experiences or data, present a different approach to the same problem, or (respectfully and collegially) argue with the presenter. In 20 hours of formal sessions, we expect to cover six to eight presentations. Some of our sessions will be activities, such as brainstorming sessions, collaborative searching for information, creating examples, evaluating ideas or work products. We also have lightning presentations, time-limited to 5 minutes (plus discussion). These are fun and they often stimulate extended discussions over lunch and at night.

Presenters must provide materials that they share with the workshop under a Creative Commons license, allowing reuse by other teachers. Such materials will be posted at http://wtst.org.

Our agenda will evolve during the workshop. If we start making significant progress on something, we are likely to stick with it even if that means cutting or time boxing some other activities or presentations.

LOCATION AND TRAVEL INFORMATION

We will hold the meetings at

Harris Center for Assured Information, Room 327

Florida Institute of Technology

150 W University Blvd

Melbourne, FL 32901

Airport

Melbourne International Airport is 3 miles from the hotel and the meeting site. It is served by Delta Airlines and US Airways. Alternatively, the Orlando International Airport offers more flights and more non-stops but is 65 miles from the meeting location.

Hotel

We recommend the Courtyard by Marriott – West Melbourne located at 2101 W. New Haven Avenue in Melbourne, FL.

Please call 1-800-321-2211 or 321-724-6400 to book your room by December 24, 2012. Be sure to ask for the special WTST rates of $93 per night. Tax is an additional 11%. All reservations must be guaranteed with a credit card by Tuesday, December 24, 2013. If rooms are not reserved, they will be released for general sale. Following that date reservations can only be made based upon availability.

For additional hotel information, please visit the hotel website at http://www.marriott.com/hotels/travel/mlbch-courtyard-melbourne-west/

OUR INTELLECTUAL PROPERTY AGREEMENT

We expect to publish some outcomes of this meeting. Each of us will probably have our own take on what was learned. Participants (all people in the room) agree to the following:

• Any of us can publish the results as we see them. None of us is the official reporter of the meeting unless we decide at the meeting that we want a reporter.
• Any materials initially presented at the meeting or developed at the meeting may be posted to any of our web sites or quoted in any other of our publications, without further permission. That is, if I write a paper, you can put it on your web site. If you write a problem, I can put it on my web site. If we make flipchart notes, those can go up on the web sites too. None of us has exclusive control over this material. Restrictions of rights must be identified on the paper itself.
  • NOTE: Some papers are circulated that are already published or are headed to another publisher. If you want to limit republication of a paper or slide set, please note the rights you are reserving on your document. The shared license to republish is our default rule, which applies in the absence of an asserted restriction.
• The usual rules of attribution apply. If you write a paper or develop an idea or method, anyone who quotes or summarizes you work should attribute it to you. However, many ideas will develop in discussion and will be hard (and not necessary) to attribute to one person.
• Any publication of the material from this meeting will list all attendees as contributors to the ideas published as well as the hosting organization.
• Articles should be circulated to WTST-2012 attendees before being published when possible. At a minimum, notification of publication will be circulated.
• Any attendee may request that his or her name be removed from the list of attendees identified on a specific paper.
• If you have information which you consider proprietary or otherwise shouldn’t be disclosed in light of these publication rules, please do not reveal that information to the group.

ACKNOWLEDGEMENTS

Support for this meeting comes from the Harris Institute for Assured Information at the Florida Institute of Technology, and Kaner, Fiedler & Associates, LLC.

Funding for WTST 1-5 came primarily from the National Science Foundation , under grant EIA-0113539 ITR/SY+PE “Improving the Education of Software Testers.” Partical funding for the Advisory Board meetings in WTST 6-10 came from the the National Science Foundation, under grant CCLI-0717613 “Adaptation & Implementation of an Activity-Based Online or Hybrid Course in Software Testing”.

Opinions expressed at WTST or published in connection with WTST do not necessarily reflect the views of NSF.