EEW Test Certification: Requests

Ticket #:
Request Date:

Ticket Status:

EEW Testing Request

Algorithm/Module Name: * required

Production Version: * required

Release Candidate Version: * required

Requester First Name: * required

Requester Last Name: * required

Requester Email: * required

Document changes between the current and the modified algorithm/module.

  - Provide a summary description of the changes, including the purpose of the changes.
  - List the specific sections of the code that were modified including any configuration changes.
  - List known defect fixes and/or enhancements associated with this build to be tested.
  - Assign defect #s or enhancement #s from Trac.
  - Copy defect or enhancement URL from Trac here.

This test must be linked to an open defect # or enhancement # from Trac. Create one if it does not exist.

* required

Summarize expected changes to performance during historic event suite tests.

* required

Summarize expected changes to performance during real-time tests.

* required

Summarize results of code review to show that it meets standard coding practice (e.g. use of coverity code review software).

* required

Confirm that algorithm/module successfully passes Unit and/or Build Validation Test (BVT) testing (e.g. does it build, compile, run without memory leaks, and run with at least one event?).

Yes
No

Additional comments. (optional)

EEW Test Plan

Algorithm/Module Name:

Production Version:

Release Candidate Version:

QA Test Plan Preparer's First Name:

QA Test Plan Preparer's Last Name:

QA Test Plan Preparer's Email:

To be completed by QA Tester

Scope: Summarize which test scenarios (e.g. historic event suite and/or real-time tests), use cases (e.g. full or subset of the historic events, individual datasets), objectives that will be validated, and requirements (e.g. dependencies, environment, new types of output, new software, hardware requirements, ability to communicate with existing architecture) based on the use cases.

Out of Scope: Summarize what will not be tested or covered (e.g. test scenarios or use cases that are not applicable).

Expected outcomes: Summarize the expected outcomes (as outlined in section 1) in order for test results to be considered successful.

Scheduling: Provide schedule with calendar dates for the documentation and aggregation of all related files (e.g. README.txt, alg documentation on wiki, makefiles, T&C test check list, past results, current results, summary report, bug fixes on Trac, test cycling).

Deliverables: List test data with associated results files that will be produced, and timeframes with calendar dates during which this will happen.

Defect and/or enhancement management: Defects/enhancements will be reported in this report and tracked in Trac. When resolved, tickets will be closed. Provide any exceptions or additions to this.

Risks and risk management: Summarize potential risks of build being tested to the production system (i.e. other ShakeAlert components), uncertainties, downstream effects, and impact on alerts.

Exit criteria: Outline criteria for if/when pass/fail occurs and how recommendation to pass or reject is made e.g. summarize how pass/fail/exit criteria are determined for results of build, installation, configuration, BVT run, historical event suite test run, two-week real-time run.

Additional comments. (optional)

QA Test Review

Algorithm/Module Name:

Production Version:

Release Candidate Version:

QA Software Builder's First Name:

QA Software Builder's Last Name:

QA Software Builder's Email:

QA Tester's First Name:

QA Tester's Last Name:

QA Tester's Email:

To be completed by QA Tester

Build/Install/Configure:

Build algorithm/module
Install & Configure algorithm/module on test replay and test real-time systems.
Run Build Validation Test (BVT) and verify build, etc.

Report any build and/or BVT issues/failures here, and also notify developer.

Test Automation:

Write test automation for new test cases, if needed.

Test Automation Comments:

Execute Test Cases:

Run historical test suite on replay.
Run real-time test systems for a minimum of two weeks.
Run additional function unit testing not included in historical replay test suite.

Test Execution Comments:

Analysis of historical test suite results:

Generate & analyze test results, and write output parameters.
Provide test run results, test analysis results, and algorithm/module logs to developers for review.

Analysis Comments:

Dev Test Review

Algorithm/Module Name:

Production Version:

Release Candidate Version:

Requester's Name:

Requester's Email:

QA Tester's Name:

QA Tester's Email:

To be completed by Developer or Ticket Requester

Provide a summary review of the performance of the defects and/or enhancements that were the focus of the test.

Provide a summary review of the algorithm/module logs from the historic event suite test provided by the T&C platform.

a. Describe any warning and/or error messages.
b. Describe picks/parameter differences between production and test version.
c. Summarize principal differences in algorithm performance for production and test version.

Provide a summary review of the algorithm/module logs from real-time run provided by the T&C platform.

a. Describe any warning and/or error messages.
b. Describe picks/parameter differences between production and candidate version.
c. Summarize principal differences in algorithm performance for production and candidate version.

T&C Summary Report

Algorithm/Module Name:

Production Version:

Release Candidate Version:

Requester's Name:

Requester's Email:

T&C Summary Report Preparer Contact Information:

Preparer's First Name:

Preparer's Last Name:

Preparer's Email:

To be completed by T&C Preparer

Historic event test suite

a. Provide Executive Summary statistics for historic event suite test comparing production to candidate version.
b. Summarize the historic event suite test results, including an assessment of whether candidate version met
the stated purpose of the proposed change and whether performance was improved.

Real-time test server (minimum two-week)

a. Summarize the statistics for real-time test server performance comparing production to candidate version.
b. Describe the real-time code performance, including computational load/overhead, potential impact on other
parts of ShakeAlert components, and an inventory of crashes and required restarts.
c. Summarize the real-time test results, including an assessment of whether the candidate version performed satisfactorily.

Deployment State

Algorithm/Module Name:

Production Version:

Release Candidate Version:

Requester's Email:

Build Deployment Engineer Contact Information:

Deployment Engineer's First Name:

Deployment Engineer's Last Name:

Deployment Engineer's Email:

To be completed by Build Deployment Engineer

Confirm ExCom approval or rejection of candidate algorithm and/or module deployment to the EEW production systems:

ExCom Approved Candidate Version
ExCom Rejected Candidate Version

Algorithm and/or Module Deployment Dates:

CI Production Servers	eew-ci1-prod1 deployed on:	eew-ci2-prod2 deployed on:
BK Production Servers	eew-bk1-prod1 deployed on:	eew-bk2-prod2 deployed on:
NC Production Servers	eew-nc-prod1 deployed on:	eew-nc-prod2 deployed on:
UW Production Servers	eew-uw-prod1 deployed on:	eew-uw-prod2 deployed on:

* If ExCom approves, change ticket state to "Close" after algorithm and/or module are deployed on all EEW systems AND a deployment date has been entered for each prod server.

Additional Comments optional: