Software Testing - Introduction
Testing is a process of evaluating a system by manual or automatic means to verify that it satisfies specified requirements or identifies differences between expected and actual results.
Testing is no longer an adjunct to the system development life cycle (SDLC), but rather a key part of it.
Why Software Testing?
* Software Testing is important as it may cause mission failure, impact on operational performance and reliability if not done properly.
* Effective software testing delivers quality software products satisfying user’s requirements, needs and expectations.
* If done poorly, it leads to high maintenance cost and user dissatisfaction.
Who participates in testing?
* Software customer
* Software user
* Software developer
* Software Tester
* Information service management
* Senior organization management
Objective of Software Tester
* To find bugs
* Find them as early as possible.
* Make sure they get fixed
What is Quality?
* Quality is defined as meeting the customer’s requirements for the first time and every time.
* Quality is much more than the absence of defects, which allows us to meet customer expectations.
Five Perspectives of quality
* Transcendental View : Quality can be recognized but not defined
* Producer View : It should conform to product specification
* Product View : is tied to characteristics of the product
* User View : should conform to user’s requirements
* Value-based view : based on how much a customer is willing to pay
Quality is the important factor affecting an organization’s long term performance and improves productivity and competitiveness in any organization
Quality Assurance Vs Quality Control
Quality assurance is a planned and systematic set of activities necessary to provide adequate confidence that products and services will conform to specified requirements and meet user needs.
* It is process oriented.
* Defect prevention based.
Quality control is the process by which product quality is compared with applicable standards and the action taken when non-conformance is detected.
* It is product oriented.
* Defect detection based.
* A particular method of doing some thing, generally involving a number of steps or operations is process
* The process that deals with the technical and management issues of software development is called Software Process
Continuous improvement cycle
* Plan (P): Device a plan. Define your objective and determine the strategy and supporting methods required to achieve that objective.
* Do (D): Execute the plan. Create the conditions and perform the necessary training to execute the plan.
* Check (C): Check the results. Check to determine whether work is progressing according to the plan and whether the results are obtained.
* Action (A): Take the necessary action. If your checkup reveals that the work is not being performed according to plan or that results are not what was anticipated. Device measures for appropriate action
* ISO International Organization for Standard
* SEI CMM Software Engineering Institute Capability Maturity Model (
* PCMM People Capability Maturity Model.
* SPICE Software Process Improvement and Capability Determination
* NIST National Institute of Standards and Technology
* DoD Department of Defense
* Six Sigma Zero Defect Orientation
* CMM level 5 Optimizing
* CMM level 4 Managed
* CMM level 3 Defined
* CMM level 2 Repeatable
* CMM level 1 Initial
Project management is concerned with the overall planning and co-ordination of a project from inception to completion aimed at meeting the client's requirements and ensuring completion on time, within cost and to required quality standards.
Needs for Project Management
Subject to Budget & Schedule it includes
* Project Planning
* Project Scheduling
* Project Costing
* Monitoring Reviews
* Report Writing & Presentation
Risk Management is concerned with identifying risks and drawing up plans to minimize their effect on a project.
A risk is a probability that some adverse circumstance will occur
Requirement Management is managing changes in the evolving software in a cost effective manner.
* CM System (Contains all the changes or versions)
* Shared (Current Version)
* Draft (Forecasted)
Automation Tools (CM) Types of Tools:
* VSS - Visual Source Safe (Microsoft Product)
* Rational Clear Case (Rational Corporation Product)
* CVS - Concurrent Version System. (Free tool)
Software configuration management plan (SCMP)
* CI - Configurable Item
* CR - Change Request
* CCB - Change Control Board
What is primary role of software testing?
* Determine whether the system meets specifications (producer view)
* Determine whether the system meets business and user needs (customer view)
A defect is a variance from a desired product attribute.
Two categories of defects are:
* Variation from product specifications
* Variation from customer/user expectation
Classifications of Defects
* Wrong - The specification have been implemented incorrectly.
* Missing - A specified or wanted requirement is not in the built product.
* Extra - A requirement incorporated into the product that was not specified.
Classification of Testing
Testing techniques that involve analysis of application components and/or processing results without the actual execution of the application. A quality assurance (QA) review is a form of static testing.
Testing, based on specific test cases, by execution of the test object or running programs.
Techniques used are determined by the type of testing that must be conducted
Functional and Structural
* Structure of the program is not considered.
* Test cases are decided based on the requirements or specification of the program or module
* Hence it is often called as “Black Box Testing”.
* Concerned with testing, the implementation of the program.
* Focus on the internal structure of the program.
* The intent of structural testing is not to exercise all the different input or output condition but to exercise the different programming structures and data structures in the program.
User Acceptance Testing
* Test each module individually.
* Follows a white box testing (Logic of the program)
* Integrate two or more modules and test for the communication between the modules.
* Follows a white box testing (Testing the code).
* Confirms that the system as a whole delivers the functionality originally required.
* Follows the black box testing.
User Acceptance Testing (UAT)
* Building the confidence of the client and users is the role of the acceptance test phase.
* It is depend on the business scenario.
White Box Testing
* Also known as glass box testing.
* A software testing technique whereby explicit knowledge of the internal workings of the item being tested are used to select the test data.
* White box testing uses specific knowledge of programming code to examine outputs.
White Box Testing Techniques
* Statement coverage – execute all statements at least once.
* Decision coverage - execute each decision direction at least once.
* Condition coverage – execute each decision with all possible outcomes at least once
Black Box Testing
A testing method where the application under test is viewed as a black box and the internal behavior of the program is completely ignored. Testing occurs based upon the external specifications. Also known as behavioral testing, since only the external behaviors of the program are evaluated and analyzed
Black Box Testing Techniques
* Equivalence Partitioning
* Boundary Analysis
* Error Guessing
A subset of data that is representative of a larger class
For example, a program which edits credit limits within a given range ($10,000 - $15,000 would have 3 equivalence classes:
* Less than $10,000 (invalid)
* Between $10,000 and $15,000 (valid)
* Greater than $15,000 (invalid)
Boundary Value Analysis
A technique that consists of developing test cases and data that focus on the input and output boundaries of a given function
In the same credit limit example, boundary analysis would test:
* Low boundary plus or minus one ($9,999 and $10,001)
* On the boundary ($10,000 and $15,000)
* Upper boundary plus or minus one ($14,999 and $15,001)
* Based on the theory that test cases can be developed based on experience of the Test Engineer
* For example, in an example where one of the inputs is the date, a test engineer might try February 29, 2001
A disciplined method of testing the interfaces between unit-tested programs as well as between system components
Incremental Testing Types
Begins testing from the top of the module hierarchy and works down to the bottom using interim stubs to simulate lower interfacing modules or programs
* Begins testing from the bottom of the hierarchy and works up to the top
* Bottom-up testing requires the development of driver modules which provide the test input, call the module or program being testing, and display test output
* A technique, often used during early integration testing
* Demonstrates key functional capabilities by testing a string of units that accomplish a specific function in the application
Verification and Validation
Verification is the process confirming that software meets its specification
Validation is the process confirming that it meets the user’s requirements
* In process Review
* Phase end/Decision Point/Milestone Review
* Post Implementation/Post Mortem
Classes of Reviews
* Informal Review (or) Peer review
* Semiformal Review (or) Walk Through
* Formal Review (or) Inspection
* Also called peer-reviews
* Generally one-to-one meeting
* No agenda (scheduling)
* Results are not formally reported
* Occur as needed through out each phase
* Facilitated by the author
* Presentation is made with comment throughout and at the end
* Reports are distributed to the participants
* Possible solutions for defects not discussed
* Occur one or more times during a phase
* Facilitated by moderator
* Assisted by recorder
* Meetings are planned in advance
* Directly depends on preparation of participants and Held at any time
Test Level Criteria –
Define the required conditions and standards that must be present for entry into the next stage of the development process.
Defines standards that blocks the promotion of the defective work to subsequent stages of the development process
Requirements Exit Criteria
* Both functional and non-functional requirements are defined
* Clear and understandable requirements
* Requirements are measurable
* Test cases are measurable
Requirements Analysis Exit Criteria
* Functional and non-functional requirements are documented
* Test cases are clearly defined and reviewed
* System test cases are validated and approved
Design Exit Criteria
* Design reviewed for testability
* It should comply with applicable standard
* Design validated for completeness, correctness and requirements coverage
Code / Unit Test Exit Criteria
* Compile, Link and Load
* 100% code coverage
* A documented evidence that all paths are executed at least once
* Adequate program documentation
Integration Entrance Criteria
* A documented evidence that all basic path is executed at least once
* Adequate program documentation
* Verify that correct unit of the system has been taken for integration
Integration Exit Criteria
* Successful execution of integration test plan
* No open defects with high severity
* Component Stability
System Entrance Criteria
* Successful execution of integration test cases
* No open defects with high severity
* 75-80% of the system functionality and 90% of major functionality are delivered.
System Exit Criteria
* Successful execution of system test cases
* A documented evidence of requirements coverage and high risk system components
* 100% of total system functionality delivered
Determines how well the user will be able to understand and interact with the system.
This is done prior to the testing levels.
Vendor Validation Testing
* This can be conducted jointly by software vendor and the team.
* Ensuring that all the requested functionality has been delivered.
* Prior to accepting it & installing it into a production.
A sanity test or "smoke test" is a brief run-through of the main functionality of a computer program or other product. It gives a measure of confidence that the system works as expected prior to a more exhaustive round of testing.
Ad Hoc Testing
Testing carried out using no recognized test case design technique.
The process of adapting software for a particular country or region. For example, the software must support the character set of the local language and must be configured to present numbers and other values in the local format.
The selective retesting of a software system that has been modified to ensure that any bugs have been fixed and that no other previously working functions have failed as a result of the reparations and that newly added features have not created problems with previous versions of the software. Also referred to as verification testing
Simulated or actual operational testing at an in-house site not otherwise involved with the software developers.
Operational testing at a site not otherwise involved with the software developers.
Testing whether the system is compatible with other systems with which it should communicate.
Benefits Realization Test
* It is a test or analysis conducted after an application is moved into production.
* To determine whether the application is likely to deliver the original benefits.
* This is conducted by the user or client group who requested the project.
* How many users a particular system can handle
* Operating the software with largest possible data files
* Repeatedly doing the load test
* Running software under less condition (low memory, low disk space)
* Testing on pull-down resource (testing external application such as server, or external hardware such as printer)
Testing life Cycle
* Test Plan A document that defines the overall testing approach.
* Test Case Design Document that defines what is selected to test and describes the expected result. Set of procedures in order to find out the defects.
* Test Execution Executing the test cases.
* Test log Pass and fail information will be stored in this test log.
* Defect tracking All the failed items will come under the defect tracking. (Closing defect)
* Test report Reports are generated during the TLC. Final test summary report. Generated at the end of testing for making decisions
Test Plan Preparation
Test Objective : A test objective is simply a testing “goal”. It is a statement of what the tester is expected to perform during the testing activity.
* Techniques that are going to be used during the testing.
* What types of tests must be conducted.
* What stages of testing are required? (E.g. unit, system)
Test Risk Analysis:
* A risk is a condition that can result in a loss. The risk that may occur/arise during the analysis of the testing.
* The probability that the negative event will occur.
Test schedule & Resources
Task Duration Resource
Test plan 3 1
Test design 5 3
Roles and Responsibilities
* Test Manager- Manages the entire testing activity (approve).
* Test Leader - Prepare the test plan, review test case, monitor defect tracking, and provide resources.
* Test Engineer - Prepare test case design (test risk, reports).
The communications during the project will be informed through this communication approach.
* Software Requirements.
* Hardware Requirements.
* Tools that are needed.
Test Case Design (manual)
* Test case ID a unique number to identify the test case
* Test case description a short note about the testing.
* Test case procedure Each and every step has to be mentioned in test case procedure.
* Test inputs (or) Test Data : Input data
* Expected result The Expected out come of the test case.
* Actual result what we have received is the actual result.
If Expected result = Actual result then the test is pass otherwise it is fail.
All the failed test execution/defects will come under the defect tracking.
* Defect Id Unique number to identify defect.
* Test case Id Test case which failed.
* Defect Description: Specify the defect.
* Defect Recovery procedure: Repeating the same test cases with some modifications.
* A metric is a mathematical number that shows a relationship between two variables.
* Software Metrics are measure that is used to quantify the software, software development resource and software development process.
Classification of Metrics
A metric used to measure the characteristic of the methods, techniques and tools employed in developing, implementing and maintaining the software system
A metric used to measure the characteristic of the documentation and code.
Use: To find the Involvement of the tester
= User Participation Test Time / Total Test Time
Acceptance Criteria Tested
Use: This metric identifies the no of user identified criteria that were evaluated during the testing process.
= Acceptance Criteria Verified / Total Acceptance Criteria
Detected Production Defect
Use: This metric show the no. of defects detected in each module.
=No of Defects detected in production /Application System size
* Server Side Interfaces
* Client side Compatibility
Objective is to check for all the links in the website.
* All Internal Links
* All External Links
* All mail to links
* Check for Broken Links
Field Level Validation
Check for length, special characters, numerical characters etc.,
Create, modify, view and delete are working.
Error handling for wrong inputs or actions.
Check whether appropriate error messages to be displayed.
Optional and mandatory fields.
Mandatory field should not be left blank. Optional should allow the user to skip
Check whether cookies are enabled.
Should not be any missing or wrong data in the database.
* Application navigation is proper through tab
* Navigation through Mouse
* Any hot keys, control keys to access menus.
Correctness is whether the information is truthful or contains misinformation. The accuracy of the information is whether it is without grammatical or spelling errors. Remove irrelevant information from your site.
* Page appearance
* Color, font and size
* Consistent design
Server Side Interfaces
* Verify that communication is done correctly, Web server-Application server
* Application server-Database server and vice versa.
* Compatibility of server software, hardware, network connections.
* Database compatibility (SQL, Oracle etc.)
Client Side Compatibility
Check for the compatibility of
* Windows (95, 98, 2000, NT)
* Unix (different sets)
* Macintosh (If applicable)
* Solaris (If applicable)
* Internet Explorer (3.X 4.X, 5.X)
* Netscape Navigator (3.X, 4.X, 6.X)
* Browser settings (security settings, graphics, Java etc.)
* Frames and Cascade Style sheets
* Applets, ActiveX controls, DHTML, client side scripting HTML specifications.
* Text and image alignment
* Colours of text foreground and background
* Scalability to fit paper size
* Tables and borders
Try with Connection speed: 14.4, 28.8, 33.6, 56.6, ISDN, cable, DSL, T1, T3
* What is the estimated number of users per time period and how will it be divided over the period?
* Will there be peak loads and how will the system react?
* Can your site handle a large amount of users requesting a certain page?
* Large amount of data from users.
* Typical areas to test are forms, logins or other information transaction components.
* Performance of memory, CPU, file handling etc.
* Error in software, hardware, memory errors (leakage, overwrite or pointers)
* Valid and Invalid Login
* Limit defined for the number of tries.
* Verify Log files are maintained to store the information for Traceability.
* Verify encryption is done correctly if SSL is used (If applicable)
* No access to edit scripts on the server without authorization.
* Time-out Occurrence