Quality Assurance Plan

From ZenWiki

Jump to: navigation, search

Contents

Revision History

Date Author(s) Changes Version
04/02/07 Session Initial draft creation 1.0
04/08/07 Session Incorporated team comments and refined the document 1.1


Review and Approval

Reviewer Version Reviewed Approval Date


Overview

Acknowledgments

We would like to thank previous studio teams Serendipity (https://msepc19.sp.cs.cmu.edu/Main_Page) and Dumbledore (http://dogbert.mse.cs.cmu.edu/mse2004/projects/Dumbledore), whose QA Plan served as a guide for writing this one.

Purpose

The purpose of this Quality Assurance Plan is to define the techniques and procedures that will be used to assure a timely delivery of the ZEN Tool that meets customer requirements and quality attributes (captured in the SRS) within the project's alloted resource constraints. This document is meant to serve as an assurance to the customer that the mentioned testing procedures will be followed, and serve as a guideline for the testing team (all members of the ZEN Team) as it performs the tests.

The goal of the testing effort is to provide the development team, at an early stage and in an iterative manner, with feedback on the accuracy and quality of their work, so that fixes are integrated early and the code is properly validated.

The accuracy and quality of the work will be determined by:

  • The percentage of requirements that are met by the system.
  • Adherence to all quality attribute scenarios described in the SRS.


The use of this plan will help ensure the following:

  • That software development, evaluation and acceptance standards are developed, documented and followed.
  • That test results adhere to acceptance standards.
  • That project documents are well maintained, baselined, and changes to them are documented (see Configuration Management Plan).


Scope

The QA plan describes quality assurance activities pertaining to the design, implementation, testing, and delivery of the ZEN Tool. The plan also defines activities related to ensuring the quality of documents developed during the development of the system (including architecture, this QA plan itself, and others).

Timeline

The testing phase is incorporated as part of the development phase in order to make it highly iterative and obtain as much feedback as possible. The timeline for test completion (when all code passes the testing process described bellow) is the end of the summer semester 2007, at which point the tool shall be complete and deployed to the client's work environment. However, particular testing activities are described in this document with their own time estimates.

Point of Contact

This document has been drafted by Session Mwamufiya, and all change requests to it (while the document is under his ownership) should be directed to him. All server access and tool operation inquiries shall be directed to the Support Engineer. By the beginning of the summer term, this document's ownership will be transfered to the Quality Assurance Manager role (which will be added to an existing team role per majority team decision), who will be responsible for keeping it up to date.

The Quality Assurance Manager in the summer is Lung-San (Allen) Hsu.

Project Overview

The SEI conducts a number of client engagements to assist clients to analyze the effort, risk, and cost to migrate specific legacy systems to specific service oriented architectures (SOA). This process is entitled the Service Migration and Reuse Technique (SMART). While it is highly valued by clients, the process is manual, and thus labor intensive and time consuming to conduct the engagement interviews and compile reports.

The ZEN Tool’s main goal is to improve the efficiency of the SMART process. Our solution, through automation, enhances three key components to achieve this objective:

  • Interview process: the migration process is streamlined with efficient data collection, automated interview guide navigation, and automated data consolidation.
  • Reporting process: reports are automatically generated from consolidated data.
  • Evolution process: engagement history is tracked and significant trends can be tracked to enhance future engagements


Related Documents List

Document name Link
Architecture click to view
Software Requirements Specification (located in the SVN)
Software Risk Evaluation (located in the SVN)
Operations Proposal click to view
Configuration Management click to view


Personnel Allocation

The Zen Development Team members are:


Legal Requirements

None.

Test Plan

Strategy

Our approach to guarantee that all must-have functional requirements and quality attributes are implemented with a high level of quality entails the following error detection and fix verification mechanisms that will be performed on an iterative basis throughout the development cycle:

  • Acceptance tests: to validate that each component fulfills its responsibilities as specified in the SRS and detailed in the Architecture, and that the client checks off on them.
  • Unit tests: to validate the correct performance of each implemented class.
  • Automated regression tests: to ensure that new code does not adversely affect existing code. This will include compiling all of the release code and running the entire bank of unit tests currently present.
  • Load testing: to gage the limits of the system (memory, data load, client load, ...) within the proposed release environment and perform appropriate changes or accept certain trade-offs.
  • Static analysis of code: to detect coding violations that are relevant to the development project, and enable the team to fix them before they become bugs or cause unpredictable logical errors. PMD will be used as the static analysis tool.
  • Bug tracking and fixing: to detect and remove any unwanted and potentially erroneous functionalities that are injected in the code.

These steps will help the development team meet its quality assurance goal of proactively addressing quality throughout the development cycle.

Process

We will assure the quality of the project by adhering to a test-driven development approach and undertaking the following iterative steps for each module at the individual developer level:

  1. Design
    1. Develop acceptance tests from use case scenarios for the component/module and confirm these acceptance tests with the client.
    2. Develop a class diagram for the module
    3. Specify API's that will be exposed
    4. Artifacts: acceptance test cases, class diagrams, and API document
  2. Design review
    1. Review the class diagrams to ensure conformity with architecture
    2. Review the interfaces exposed, and document component dependencies
    3. Artifacts: review comments and developer response (conformed to, or justification why not) will be stored in a issues tracking tool.
  3. Code
    1. Develop unit tests, and ensure that the code will be at least 60% covered.
    2. Implement the module's required responsibilities based on the Architecture document.
    3. Fix all errors.
    4. Artifacts: code files with comments
  4. Verification
    1. Run the unit tests (ensure that all tests pass).
    2. Run PMD using the following rulesets and fix all non-trivial warnings. Rulesets: basic, clone, codesize, design, imports, strictexception, strings and sunsecure.
    3. Artifacts: PMD test reports
  5. Validation
    1. Run the acceptance tests (ensure that all acceptance criteria are met).
    2. Artifacts: Acceptance test reports
  6. Code review
    1. The review process is defined here.
    2. Artifacts: review comments and developer response (conformed to, or justification why not) will be stored in a issues tracking tool.

At the build level, the following will happen for each build:

  1. Regression tests will be run for each build of the system.
  2. All bugs discovered (at any time) will be documented with their reproduction steps for future fixing.

At the build level, in addition to the use case validation and unit tests, stress/load tests (described below) will also be run to test the operational limits of the system if that need arises in the future.

At the overall system level, before every client release:

  • Integration testing will be performed.

Testing priorities and focus

We will be using a test-driven development approach, and be conscious of quality throughout the development cycle by following our QA Plan and CM Plan, in order to ensure that development guidelines (architecture, proposals, …) are correct and up to date, and that the implementation conforms to these guidelines. If the module's architectural design is altered, then the implementation must be modified in turn to reflect these changes. Given that our testing plan will be executed in close collaboration, and almost as part of, our development plan, priority during testing will be attributed to the modules being developed at any point in time. There will be a focus on automating the testing activities where possible.

Testing mechanisms

Acceptance Testing

Each module is described in the architecture document with a specific list of responsibilities built from use cases and requirements; the acceptance test for a given module will consist of this list of responsibilities, and the client will agree on the test criteria for each component. Upon each release of the system, the client will be presented with the components of the release and the acceptance test results for each component. The exit criteria will be that the client agrees with the acceptance test results.

Unit Testing

All modules will be unit tested to ensure that the individual unit (class) performs the required functions and outputs the proper results and data. Proper results are determined by using the design limits of the calling function as specified in the design specification defining the called function. Unit testing is typically white box testing and helps ensure proper operation of a module, because tests are generated with knowledge of the internal workings of the module. Unit tests will be implemented and run by a module's owner. The exit criteria will be that all unit tests pass. Unit tests should cover the following:

  • Correct Functionality: Build tests that validate that the code does what it's supposed to do.
  • Boundary Analysis: Specify tests that will execute code using boundaries at n-1, n, n+1. This includes looping instructions, while, for and tests that use LT, GT, LE, GE operators.
  • Error handling: Design tests that verify the recording of all detected and reportable errors that the program is designed to find and report.
  • Mathematical limit checking: Design tests that use out of range values that could cause a mathematical function to calculate erroneous results.


Regression Testing

The purpose of regression testing is to rerun previous tests and make sure that they are still valid, thus confirming that new code development does not adversely affect the existing code base. Our regression tests will consist of a bank of unit tests that have been run for all modules present in any given build. We will be using JUnit, as it provides us with the ability to run unit tests as part of a bank of tests. Regression testing will be automated, and the Quality Assurance Manager will document errors and report them to the owners of the modules involved. The exit criteria will be that no new errors in the previous build our found when running the regression tests.

Integration Testing

The integration testing is meant to ensure that all modules within a release version of the system are inter-operating properly and in accordance to the client's expectations (as described in the acceptance tests that she agrees upon for each component). This helps ensure that the system as a whole adheres to the client's expectations. Integration testing will be performed by the Quality Assurance Manager each week; he will document errors and report them to the owners of the modules involved. The exit criteria will be that all component acceptance tests pass after the components are integrated with each other.

Load Testing

We do not currently have any capacity requirements for the tool, but if some should arise, once the server is ready to handle client requests, we would run tests using multiple clients to make sure that the system can handle the required load (and more, to take into consideration future scalability). The results of stress tests would be identified by the corresponding build number and recorded for each build. The exit criteria would be that all load tests are run and their results documented (there may not be any code fixes required, and may lead us to rethink the architecture design if important system limitations are discovered).

Bug tracking and resolution

Bugs discovered in the builds will be documented in detail in our bug tracking tool, TBD, with a set of reproduction steps whenever available. Each bug found will contain the following information:

  • Title
  • Related build id
  • Developer who discovered it
  • Description
  • Reproduction steps
  • Assigned developer to fix the bug


Each Developer will devote half a day per week during the summer term to fix outstanding bugs. This frequency will changed depending on the number of outstanding bugs and the Team Lead's prerogative.

Once a bug is fixed, the fixing developer should note the following information in the bug tracking tool:

  • Source of bug
  • Brief description of fix
  • Updated bug status


The Chief Scientist will track the bug trends, and will note the point at which the team first reaches zero defects after the bug discovery and resolution efforts converge.

Problem tracking and resolution

Separately from the code related bugs, problems and issues with the project will be tracked and followed separately by the team lead, who will be responsible to make sure that they get resolved appropriately.

Standards

Coding Standards

The ZEN Team's coding standards can be found at Coding Standards.

Naming Conventions

TBD

Test site/location

All tests will be performed and logged on the team's virtual server.

Test environment setup and configuration issues

All issues with the testing and development environments should be directed to the Support Engineer.

Development Environment

The following tools will be available to developers on their personal development environments:

  • Eclipse
  • JUnit
  • SVN client
  • MediaWiki
  • MySQL
  • PMD (for static analysis)
  • Eclemma (for test coverage)

The following tools will be available on the team's development server:

  • SVN server
  • MySQL
  • Ant (for task automation)
  • CruiseControl (for nightly builds)
  • JUnit (for unit and regression testing)
  • PMD (for static analysis)
  • Bugzilla (for bug tracking)
  • Emma (for test coverage)

Reviews

Document Review Process

The architectural review process will be as follows:

  1. The document owner updates the document.
  2. The document owner assigns at least 1 reviewer to validate the document update based on the document's review guidelines.
  3. The reviewers follow the document's review guidelines and communicate their findings to the document owner within 3 days.
  4. The document owner updates the document with the reviewer(s)'s comments within 2 days.
  5. The document owner does a run through of the document with the entire team and notes down additional todo items (if any).
  6. The document owner updates the document with the team's comments within 2 days.
  7. The document owner makes the document available to the mentors for their comments.
  8. Mentors provide feedback within 4 days.
  9. The document owner updates the document with the mentors' feedback within 2 days.
  10. The document owner presents the document to the mentors for a final review.
  11. Mentors approve the state of the document, or provide additional feedback to the document owner, who will iterate through the last two steps.


Architectural Reviews

Every developer will be invited to all design reviews throughout the entire software development life-cycle to ensure peer reviews of the software design are conducted.

Updates to the architecture document will undergo the Document Review Process steps with the Architecture Review Guidelines. The architecture itself will be reviewed periodically for completeness and soundness using the following approach:

  1. Architecture view validation:
    1. Ensure that the appropriate viewtype is used to document the given part of the architecture.
    2. Ensure that all elements in the viewtype are described.
    3. Detail the issues discovered from the architecture.
  2. Architectural analysis:
    1. Perform a mini-ATAM and record its results based on the QA scenarios being covered by the given part of the architecture.
  3. Rationale:
    1. Review the key architectural decisions made (including alternatives, if any).
    2. Ensure that the rationale for making architectural decisions is still sound.


The Chief Architect will identify all action items generated during this review process, and will ensure (track) that all action items have been addressed.

Code Reviews

Code reviews will be performed to review the code of the modules that have been completed.

The code review process will be as follows:

  1. Team members will make themselves available to review component code on Tuesdays and Fridays.
  2. A team member who has completed a module will request a code review.
  3. An available team members will respond to the request, and if nobody is available to review the code, the team lead will assign a team member to review the code.
  4. The reviewer will request a design walk-through from the developer if needed.
  5. The reviewer will validate the code using the code checklist (defined bellow).
  6. The reviewer will send his comments to the developer.
  7. The developer will update the component with the comments and make the chief scientist aware of the completed review.
  8. The Chief Scientist will update the status of the component.


Reviewing teams will use the following checklist:

  • Are classes/methods documented?
  • Are class members given the right access scope (private vs. public)?
  • Are variables checked for null before usage?
  • Are possible exception conditions trapped?
  • Are for-loops variables exceeding index ranges?
  • Are synchronized methods being used consistently to avoid dead-lock?
  • Are all methods' functionalities met?
  • Are there any logical errors?
  • Are there typos or problems with text messages displayed to the user?
  • Are there opportunities to enhance performance and modifiability of the code?
  • Are there descriptive comments to explain code that implements complex logic.
  • Are there possible exception not being handled?
  • Is the architecture design being followed by the implementation?
  • Are formatting conventions being followed? Note that coding convention can be automatically enforced by code format (Ctrl+Shift+F) functionality in the Eclipse IDE, which uses a Java code template file that can be customized by the team.
  • Are there JUnit test cases that could be implemented in addition to the ones that exist?

The checklist is subject to change as the team adapts its testing practices to better suit the project. The change management process for the checklist (as well as any element in the QA Plan) is defined in the Configuration Management Plan.

Software CM processes

Please refer to the Configuration Management Plan.

Training

As part of the MSE program, every member of the team has received the required training to conduct all of the types of tests described in this document. However, as the document is updated, new techniques may be brought forth that would require some knowledge transfer. In this case, the training process elaborated in the Operations Proposal will be followed to share the appropriate knowledge required.

Project test metrics

TBD

Initial test benchmark

The project will be baselined during the first week of the summer semester, in which a skeleton system will be added as the starting code base, and bugs (if any) will be recorded. During this period, all relevant project test metrics will be baselined as the starting point of testing.

Test suspension and restart criteria

Testing could be suspended due to any of the following:

  • Project metric out of control: in case the team determines a project metric (i.e the number of bugs found to bugs fixed ratio) is out of control, the team may decide to stop testing in order to perform the appropriate tasks to bring that metric back under control.
  • Architecture overhaul: if the architecture needs to be significantly altered, then the team will focus all of its resources on the architecture, and may decide to forgo a number of testing activities during that period of time. However, the automated tests may still proceed if they are deemed useful.
  • Unavailability of testing environment: if the team server goes down for some reason, the team may be forced to suspend certain testing activities (i.e. regression testing of a build) until the server and its applications are back and running again, or an alternative method is used to integrate the code.


Project migration process

Once the system has been fully testing and accepted within the testing environment by the client (through the acceptance tests mentioned above), the team will migrate the server part of the ZEN Tool onto the client's server and handle all integration and compatibility issues. The client part of the ZEN Tool will be installed on a laptop at the SEI and be tested for full functionality within a simulated engagement environment.

Both parts of the ZEN Tool will be provided to the client in the form of a CD, along with a document describing the content of the source code and any known issues.

Change Management

Major architectural changes and client requests will undergo the change management process illustrated in the Operations Proposal.

Traceability Requirements

TBD

Assumptions

TBD

Project risk analysis

The risk analysis process is described in the Operations Proposal, and it is documented in the SRE document in SVN.

Reflection

A reflection on how well the QA Plan is being followed and how useful it is to the ZEN Team will be recorded by the Quality Assurance Manager, each iteration, as part of his/her team reflection exercise.

References



Retrieved from "http://msesrv1d.mse.cs.cmu.edu/wiki/index.php/Quality_Assurance_Plan"
Personal tools