|
|
| Home > Software > software-eng > |
comp.software.testing Frequently Asked Questions (FAQ) |
Section 1 of 2 - Prev - Next
Archive-name: software-eng/testing-faq
Posting-Frequency: monthly
Last-modified: 2002/08/06
Version: $Revision: 3.9 $
Maintainer: cst-faq@crim.ca
URL: http://www.faqs.org/faqs/software-eng/testing-faq/
http://www.crim.ca/ctl/cst.FAQ.html
http://www.cigital.com/c.s.t.faq.html
comp.software.testing FAQ
and information resource
This is the Frequently Asked Questions list for comp.software.testing.
This FAQ is cross-posted to comp.software-eng. Please don't hesitate
to send corrections, additions, and other comments to
cst-faq@crim.ca.
Be sure to use the tag of FAQ: in the subject line, for example,
"Subject: FAQ: Modification request...."
The latest version of this FAQ may be retrieved using Usenet, the World
Wide Web, ftp, email, gopher, or even telnet. See the FAQ "Introduction
to the *.answers newsgroups" in news.announce.newusers or
for details. Please check the date above - if this file is more than a
month old, it is obsolete. The main ftp site for the FAQ is
.
The folks at CRIM Software Test Center sponsor a web site
for the FAQ at http://www.crim.ca/ctl/cst.FAQ.html.
The folks at Cigital sponsor a web site for the FAQ, at
.
This file is Copyright 2001, 2002 by Raymond Rivest. Danny R. Faught is the
original author. Permission is granted to distribute this file unedited
and in its entirety, provided the "Date" header is no more than one month
earlier than the date of distribution. This information is provided
without any express or implied warranty.
------------------------------
Subject: 1. Table of Contents
1. Table of Contents
2. Change Log
3. What is this newsgroup about?
4. Netiquette
5. Beta testing
6. What other FAQs are relevant?
7. What other resources are available?
8. World Wide Web resources
9. What's the difference between QA and testing?
10. I'm looking for a test tool...
11. How do I find information about testing object-oriented programs?
12. How do I test web-related applications?
13. What is black box/white box testing?
14. What are unit, component and integration testing?
15. What's the difference between load and stress testing ?
16. Should we discuss bug tracking tools in this newsgroup?
17. What kind of salary should a tester make?
18. Where can I find sample test plans?
19. What is the best tester to developer ratio?
20. Conferences
21. Periodicals
22. Books
23. Bibliographic resources
24. Organizations
25. Contributors
------------------------------
Subject: 2. Change Log
This section will highlight notable additions, deletions, and changes to
the FAQ.
Changed in version 3.:
o Some minor cosmetic changes.
o Added CRIM STC in section 24.
Changed in version 3.8:
o Added Extreme Programming mini-FAQ link in section 6.
Changed in version 3.7:
o Finally cleaned up Danny's e-mail to tejasconsulting :)
o Removed "The Outlook" is no longer available from McCabe? (subject 22).
o Added a commercial newsletter to subject 22: The Payne Report,
http://www.cigital.com/paynereport/.
o Changed the comp.software.config-mgmt url to
http://www.daveeaton.com/scm/PMTools.html.
o Added http://www.QALinks.com and http://www.TestingFAQs.org
in subject 8.
o Added Die Software Testing Website http://www.softwaretesting.de
in suject 8.
o Added LogiGear web page at http://www.qacity.com/ in subject 8.
o Added Tejas Software Consulting Newsletter in subject 21.
o Added a new subject tag in section 4 (netiquette) for job searcher (CV:).
o Added a link to http://www.xprogramming.com/software.htm
in subject 7 about eXtreme programming and testing.
o Added two references about security testing in subject 7.
o Added Quality Web Systems by Elfriede A. Dustin/Douglas McDiarmid
in subject 12.
o Updated a broken link for http://www.asq.org/pub/ in subject 21
o Added a quick reference defining what is a "smoke test" in subject 7.
Changed in version 3.6:
o Changed reference from www.dejanews.com to groups.google.com.
Google acquired dejanews !
Changed in version 3.5:
o Inserted a new Section 15 and moved the rest downward.
What's the difference between load and stress testing ?
Changed in version 3.4:
o Reordered the Table of Content. Section 7 to 4, 10 to 5 and moved the
rest downward to explain the purpose of comp.software.testing from
the start.
o Changed references of Danny Faught's FAQs from his home page to
http://www.testingfaqs.org/ in Section 10 (a forgotten link).
o Added TESTCOM conference to Section 19.
o Added reference to the defect tracking tool FAQ in Section 15.
o Added StickyMinds.com in Section 8.
o Added a book reference in Section 12.
o Added the StickyMinds.com template section in Section 17
o Added a reference to comp.software.measurement newsgroup in
Section 7.
Changed in version 3.3:
o Added the mention of subject tag for cst-faq alias because of spams.
o Changed references of Danny Faught's FAQs from his home page to
http://www.testingfaqs.org/
o Added reference to BetaSoft's QAForums in section 5.
Changed in version 3.2:
o Changed references of Danny Faught's FAQs to his home page.
o Changed Danny Faught's email.
o Added LatinStar 2001 conference.
o Changed STQE booklist to stickyminds.
o Added a new URL for the html FAQ.
Changed in version 3.1:
o Changed references of cst-faq@cigital.com to cst-faq@crim.ca.
Changed in version 3.0:
o Head FAQ maintainer responsibilities transferred from Danny Faught to
Raymond Rivest. Have fun with it, Raymond!
Changed in version 2.7:
o Delegated some items to Cigital's SRM Hotlist, so I don't have to
maintain them in two different places. This includes the list of mailing
lists (subject 5), web pointers and dates for conferences (subject 19),
and the entire book list (subject 21). This will also help to make the
FAQ somewhat less bulky.
o Removed the Triangle Quality Council, which is being phased out.
Also removed Association for Software Testing Australia, which no
longer seems to exist. (subject 23)
Changed in version 2.6:
o Reliable Software Technologies is now Cigital, Inc. Changed all
RST references to Cigital.
o Reflected the change from American Programmer to Cutter IT Journal.
(subject 20)
o Removed information about the cst-improve list, which is no longer
in use. (subject 5)
o Updated contact information for Software Practitioner. (subject 20)
------------------------------
Subject: 3. What is this newsgroup about?
If you're new to Usenet, please read through the FAQs in
news.announce.newusers and hang out in news.newusers.questions for a
while before you consider posting.
If you do not know how to subscribe to comp.software.testing, you
first need to get Usenet access from an Internet service provider or
your company, and you need newsreader software. Different service
providers support different newsreaders, so there is no general
answer. Ask your service provider or local system administrator to
help. If all else fails, use a web browser to access Usenet at
.
The original charter for comp.software.testing can be found at
. Here's an
excerpt (with corrections):
CONTENTS:
For the purpose of this discussion, we will assume that a system is
composed of hardware, software, people, and procedures. The
proposed group should be chartered to include discussions
characterized by the following list of subjects, compiled from
preliminary discussions:
system test automation
system test planning
system test optimization (e.g. minimize regression testing)
test tools
test metrics
measurement technology
nature of testing under various development models
(e.g. object-oriented, real time, etc...)
testing in a rapid prototyping environment (i.e. sans spec)
relationship of various classes/types of tests to requirements, ...
conference and symposium announcements
the definition of "software testing" and its relationship to
SQA and debugging.
requirements traceability
risk-based testing
the most useful form of specifications and functional
requirements from the tester's point of view.
testing techniques, e.g., structured testing using control
flowgraphs and basis path testing, equivalence class
partitioning, boundary value analysis, cause-effect graphing,
path predicate testing, data flow testing, program slices,
data object state and usage analysis, data flow anomaly
analysis, and sensitivity analysis.
test coverage criteria, e.g., statement (C0), path, branch (C1),
module (S0), and call-pair (S1) coverage criteria.
All levels of testing for both hardware and software should be
considered appropriate subject matter. While it is likely that the
group will take on a software flavor early on, we should try to
generate interest and input from people with hardware and system
testing experience and perspective. It is especially important that
we foster discussion of *system* level testing issues, since this
is the weakest area of our collective knowledge.
------------------------------
Subject: 4. Netiquette
Did I mention that you should read the FAQs in news.announce.newusers if
you haven't done so lately? Also, you can reduce the chances of
embarrassing yourself by reading comp.software.testing for a while
before posting. At the very least, scan all the articles in the group
currently on your server to make sure your question hasn't already been
asked and answered.
Please help the newsgroup's participants find articles in the categories
they're interested in. If your posting falls into the categories below,
please put the three-letter tag at the beginning of your subject line.
This will also increase the likelihood that those wanting to read your
post will find it. For example, your subject line might read -
"Subject: ADV: Widget Tester 6.0 released".
JOB: Help wanted ads
ANN: Announcements for new books, publications
CFP: Call for papers or participants
CNF: Conference announcements
ADV: Other advertisements and commercial product announcements
CV: Curriculum Vitae, looking for a job
Job postings that are cross-posted to the various job-related
newsgroups should not be posted to comp.software.testing, since the
subject conventions for those groups do not allow them to be easily
identified as job postings, and the misc.jobs FAQ
()
specifies that job postings should not be cross-posted between the
job-related newsgroups and non job-related newsgroups.
Some recruiters refuse to follow the job posting policy. If you don't
want to read job postings, consider using a kill file to
systematically kill articles from these recruiters.
Many questions about a particular test tool only interest those readers
who use that tool. Please make sure to put the name of the test tool
in the subject line so readers can select the articles they read by
looking at the subject. In general, please use a descriptive subject
line.
If you post an article and later decide you shouldn't have posted it,
please cancel it. It is much better to correct the problem than to
send *another* post to apologize for the first one. Situations where
this would be appropriate include posting empty articles, multiple
copies of the same article, and any goof where you'd like to correct
something you posted shortly after it was sent. Look in your
newsreader documentation for help with cancelling an article; some also
have a "supercede" feature which makes it easy to correct a posting.
While the cancel feature has been common for quite some time, some
newer newsreaders unfortunately do not offer it. If this is the case,
complain to the author, and consider using a more reasonable
newsreader. Here's a manual method to cancel an article - start to
follow up to the errant posting. Change the subject to
"cmsg cancel ", where message-id is taken from the
References: header or the attribution. The body of the message is not
important. Post it, and the news system should interpret it as a
cancel request rather than a real posting. You can only do this for
an article you posted yourself.
People have also expressed concern about postings that are totally
off-topic. These postings are typically "spam" postings that go to most
of the thousands of newsgroups that exist. There is very little that an
unmoderated group can do proactively to prevent them. See
news.admin.net-abuse.usenet and
for more
information.
Please don't post test messages to comp.software.testing. This group
is for discussing software testing, not for testing your news
software. If you want several automatic confirmations that your post
worked, post a test message to misc.test. There are other test groups
at various levels of the news hierarchy that all end in ".test". It's
best to try a local one first.
------------------------------
Subject: 5. Beta testing
Discussions about managing a beta test program are within the scope of
this newsgroup. However, companies looking for beta testers, and
aspiring beta testers looking for something to test, should go to
comp.sources.testers instead. A possible exception is test tool
vendors looking for beta testers.
If you are looking for a career in software testing,
comp.software.testing will welcome you. Just don't call it "beta
testing". Beta testing is typically conducted by end users of a
software product who are not paid a salary for their efforts.
------------------------------
Subject: 6. What other FAQs are relevant?
Danny Faught maintains the following FAQs, originally created by Brian
Marick:
Testing Contractors and Consultants List
Testing Courses List
Testing Tool Supplier List
These FAQs are available at .
Rick Hower maintains the "Software QA/Test Resource Center" at
. It covers a broad range of
questions about software testing.
Related to new trends, eXtreme Programming, here's the mini-FAQ:
.
------------------------------
Subject: 7. What other resources are available?
The public comp.software.testing archives and the mailing list
gateway are no longer available. As an alternative, try Google groups at
Comp.software.testing articles are archived back to March, 1995.
If you're interested in quality measures (metrics), than
will interest you. However, I didn't
find any FAQ for this newsgroup.
If eXtreme testing is of interest, than
will interest you.
See also Ronald E. Jeffries's article at
http://www.cigital.com/hotlist/publications-ml.html>.
BetaSoft hosts Web forums on most popular testing tools and techniques
at
A quick reference explaining what is a "smoke test" can be found at
.
------------------------------
Subject: 8. World Wide Web resources
Here are a few web sites that each try to connect you to much of the
software testing and quality information that is available on the web.
The Cigital Software Risk Management Hotlist
Danny Faught's testing faqs page
Die Software Testing Website
This website has lots of articles, book reviews, software testing related
software for download and a discussion forum. Most of the website is in
german language. All contents can be accessed free of charge.
LogiGear's QACity.Com : Resources for Busy Testers
.
------------------------------
Subject: 10. I'm looking for a test tool...
Did you read the Testing Tool Suppliers FAQ at
Courses on testing object-oriented code (one by Robert Binder,
one by Ed Berard) are described in the Testing Courses FAQ. Also,
a new addition is an offering from Software Quality Engineering.
------------------------------
Subject: 12. How do I test web-related applications?
Rick Hower maintains a web page which includes a nice list of web
testing tools, and some guidelines for testing web sites.
Please contact him at
rickhower@earthlink.net if you can offer additions or corrections to the
list.
One book that can be of help, "Testing Applications
on the Web" by Hung Q. Nguyen and "The Web Testing Handbook"
by Steven Splaine and Stefan P. Jaskiel.
One recent addition is Quality Web Systems by Elfriede A. Dustin/Douglas McDiarmid.
It also contains a Tool Evaluation appendix, comparing the
major tools (Mercury, Rational, Empirix, Compuware, Segue).
This matrix consists of approx. 30 pages of detailed
evaluation criteria.
------------------------------
Subject: 13. What is black box/white box testing?
Black-box and white-box are test design methods. Black-box test design
treats the system as a "black-box", so it doesn't explicitly use
knowledge of the internal structure. Black-box test design is usually
described as focusing on testing functional requirements. Synonyms for
black-box include: behavioral, functional, opaque-box, and
closed-box. White-box test design allows one to peek inside the "box",
and it focuses specifically on using internal knowledge of the software
to guide the selection of test data. Synonyms for white-box include:
structural, glass-box and clear-box.
While black-box and white-box are terms that are still in popular use,
many people prefer the terms "behavioral" and "structural". Behavioral
test design is slightly different from black-box test design because
the use of internal knowledge isn't strictly forbidden, but it's still
discouraged. In practice, it hasn't proven useful to use a single test
design method. One has to use a mixture of different methods so that
they aren't hindered by the limitations of a particular one. Some call
this "gray-box" or "translucent-box" test design, but others wish we'd
stop talking about boxes altogether.
It is important to understand that these methods are used during the
test design phase, and their influence is hard to see in the tests once
they're implemented. Note that any level of testing (unit testing,
system testing, etc.) can use any test design methods. Unit testing is
usually associated with structural test design, but this is because
testers usually don't have well-defined requirements at the unit level
to validate.
Definitions of these terms can be found in these references:
Beizer, _Black Box Testing_, p. 8.
Beizer, _Software Testing Techniques_, pp. 10-11.
Daich, et al., _STSC Software Test Technologies Report_, pp. 198, 206.
Kaner, et.al., _Testing Computer Software_, Second Edition, pp. 41-43.
Mosley, _The Handbook of MIS Application Software Testing_, pp. 68-76.
Myers, _The Art of Software Testing_, pp. 8-11.
Wilson, _Unix Test Tools and Benchmarks_, pp. 298, 306
------------------------------
Subject: 14. What are unit, component and integration testing?
The following definitions are from a posting by Boris Beizer on
the topic of "integration testing" in the c.s.t. newsgroup.
The definitions of integration tests are after Leung and White.
Note that the definitions of unit, component, integration, and
integration testing are recursive:
Unit. The smallest compilable component. A unit typically is the
work of one programmer (At least in principle). As defined, it does
not include any called sub-components (for procedural languages) or
communicating components in general.
Unit Testing: in unit testing called components (or communicating
components) are replaced with stubs, simulators, or trusted
components. Calling components are replaced with drivers or trusted
super-components. The unit is tested in isolation.
component: a unit is a component. The integration of one or more
components is a component.
Note: The reason for "one or more" as contrasted to "Two or
more" is to allow for components that call themselves
recursively.
component testing: the same as unit testing except that all stubs
and simulators are replaced with the real thing.
Two components (actually one or more) are said to be integrated when:
a. They have been compiled, linked, and loaded together.
b. They have successfully passed the integration tests at the
interface between them.
Thus, components A and B are integrated to create a new, larger,
component (A,B). Note that this does not conflict with the idea of
incremental integration -- it just means that A is a big component
and B, the component added, is a small one.
Integration testing: carrying out integration tests.
Integration tests (After Leung and White) for procedural languages.
This is easily generalized for OO languages by using the equivalent
constructs for message passing. In the following, the word "call"
is to be understood in the most general sense of a data flow and is
not restricted to just formal subroutine calls and returns -- for
example, passage of data through global data structures and/or the
use of pointers.
Let A and B be two components in which A calls B.
Let Ta be the component level tests of A
Let Tb be the component level tests of B
Tab The tests in A's suite that cause A to call B.
Tbsa The tests in B's suite for which it is possible to sensitize A
-- the inputs are to A, not B.
Tbsa + Tab == the integration test suite (+ = union).
Note: Sensitize is a technical term. It means inputs that will
cause a routine to go down a specified path. The inputs are to
A. Not every input to A will cause A to traverse a path in
which B is called. Tbsa is the set of tests which do cause A to
follow a path in which B is called. The outcome of the test of
B may or may not be affected.
There have been variations on these definitions, but the key point is
that it is pretty darn formal and there's a goodly hunk of testing
theory, especially as concerns integration testing, OO testing, and
regression testing, based on them.
As to the difference between integration testing and system testing.
System testing specifically goes after behaviors and bugs that are
properties of the entire system as distinct from properties
attributable to components (unless, of course, the component in
question is the entire system). Examples of system testing issues:
resource loss bugs, throughput bugs, performance, security, recovery,
transaction synchronization bugs (often misnamed "timing bugs").
------------------------------
Subject: 15. What's the difference between load and stress testing ?
Boris Beizer says:
> One of the most common, but unfortunate misuse of terminology
>is treating "load testing" and "stress testing" as synonymous. The
>consequence of this ignorant semantic abuse is usually that the system
>is neither properly "load tested" nor subjected to a meaningful stress
>test.
>
>1. Stress testing is subjecting a system to an unreasonable load
>while denying it the resources (e.g., RAM, disc, mips, interrupts,
>etc.) needed to process that load. The idea is to stress a system to
>the breaking point in order to find bugs that will make that break
>potentially harmful. The system is not expected to process the
>overload without adequate resources, but to behave (e.g., fail) in a
>decent manner (e.g., not corrupting or losing data). Bugs and failure
>modes discovered under stress testing may or may not be repaired
>depending on the application, the failure mode, consequences, etc.
>The load (incoming transaction stream) in stress testing is often
>deliberately distorted so as to force the system into resource
>depletion.
>
>2. Load testing is subjecting a system to a statistically
>representative (usually) load. The two main reasons for using such
>loads is in support of software reliability testing and in
>performance testing. The term "load testing" by itself is too vague
>and imprecise to warrant use. For example, do you mean representative
>load," "overload," "high load," etc. In performance testing, load is
>varied from a minimum (zero) to the maximum level the system can
>sustain without running out of resources or having, transactions
>suffer (application-specific) excessive delay.
>
>3. A third use of the term is as a test whose objective is to
>determine the maximum sustainable load the system can handle.
>In this usage, "load testing" is merely testing at the highest
>transaction arrival rate in performance testing.
------------------------------
Subject: 16. Should we discuss bug tracking tools in this newsgroup?
Discussing bug tracking tools is not within the original charter of
comp.software.testing. The comp.software.config-mgmt newsgroup seems
to have taken this topic under its wing. That group has a FAQ for
problem management tools. See the "What other FAQs are relevant?"
section for details.
Danny Faught maintains a FAQ under is wing for defect-tracking tools
at http://www.testingfaqs.org/t-track.htm.
------------------------------
Subject: 17. What kind of salary should a tester make?
Salary information is available at
and . A nice
summary of the various surveys that are available is at
Section 1 of 2 - Prev - Next
| Back to category software-eng - Use Smart Search |
| Home - Smart Search - About the project - Feedback |
© allanswers.org | Terms of use