Testing Process

By: Dan Stewart
Created: April 11, 2016
Updated: March 04, 2023

Here is my testing process.

Overview

My testing process is made up of:

  1. Test Notes
  2. Test Basis
  3. Test Plan
  4. Test Suite
  5. Test Case

The test notes are my best friend. They prove that I tested. They show how I tested, and they help me improve my techniques as a tester. The test basis is often called the oracle of truth. It is the foundation from which I build my tests. The test plan gives me the when, how, and what I will test. The test suite is a collection of test cases that I will perform.

Test Notes

When someone asks me to test something, the first thing I do is start my notes. I want to write down who asked me to test, when I can test, when I will be done, what will be tested, and how I will get paid. These notes can be copied and pasted into the test plan later.

Test Basis

I prefer calling it the test basis as opposed to the oracle. At my work, Oracle® is software, not a source of truth. I look for documentation, statutes, policies, and procedures. Software is written to enforce these policies. When a policy changes, the software has to change. When software is changed, testers make sure the change agrees with the new policies. They perform regression testing to make sure that existing policies are still enforced.

Sometimes, I get to write the policy because a question arises before the testing begins. The exciting thing about writing a policy is that it will outlive the software. I created it based on conversations I had with subject matter experts. I contacted the stakeholders and had them agree to the policy. Once they agreed, the developers enforced it in the software. I discovered a policy, got it approved, and saw it in development.

I see policies as the foundation of my testing. If I have a question, I get an answer. Then I document it and test for it. I am finding the truth. It is very rewarding.

The truth is rarely pure and never simple.

Some say, "there is no truth." To them I ask, "is that true?"

I believe there is truth. If what we thought was true changes, the truth does not change. Truth cannot change or else it would not be true. Only our understanding of the truth has changed. When I test, my goal is to reach an acceptable level of truth. That acceptable level is dependent upon the time, knowledge, and consequences that a lie would have on others.

Test Plan

My test plan includes:

Test Suite

I am seeking the truth when I test. Scientists seek truth using the scientific method. A test suite asks a question. To answer this question a hypothesis is formed. Each hypothesis is a test case because a hypothesis can be either proven or disproven. Testers predict that the software will fail. They go in with the belief that the software will not work so that they are looking for the way it will fail. It is a state of mind. The testing begins. Observations are made and carefully recorded. Experiments are conducted and scripted so that the results are repeatable. An analysis is made to reveal the answer to the question. The hypothesis is proven or disproven. Another question is formed and the process begins again.

Let me show you.

Shack Data Entry

(Link to the test basis)

Administrators need to enter data on a shack. Some of the data is required. Some of the data is validated. If these validations are broken, data corruption can occur. Shacks could be built without roofs. Renters could move into a shack without a front door. Panic, riots, unicorns dancing in the street. Is the data entry form working correctly?

Preconditions

Required fields:

Validated Fields

State Transition Testing

The life of a form is created, read-only, updated, and deleted (CRUD).

The form has two states, valid and invalid. The form should be saved when valid and not saved when invalid. Invalid forms can be made valid and saved.

Decision Table Testing

Description State
Empty form Invalid
Missing required field Invalid
Invalid field Invalid
Invalid form made valid Valid
Minimum required fields entered Valid
All fields with valid data Valid

After saving the form, do something to test that it is actually saved. I have seen things look like they were saved, but it was just client-side.

Test Case

A test case should only have one reason to fail. In the test suite above, there are a lot of test cases. Do we need to write each one? I tried it once and spent so much time writing test cases I never tested.

Automation to the rescue? I certainly believe we can automate data entry forms. Then we can verify that the form did not save when empty and did save when filled out. How do we automate that the labels are spelled correctly? Are the required fields indicated in some way? There are some test cases that can be 100% automated, others only 50% and some not at all.

I rely on the test suite to provide test cases in the form of state diagrams and decision tables. I do not want to write all of the test cases.

I use testing mnemonics and heuristics like:

Bug Reporting

The purpose of testing is to find bugs. Follow the Grice Maxims when you find a bug.

Conclusion

Let me close with this. I will write notes, basis, plans, and suites. I will code test cases or perform them manually. Writing test cases is tedious, hard to maintain, and not of much value. I prefer to lean on my ability to test instead of documenting every possible test case. To illustrate this point, here are some test cases for this form. I believe these should be in the mind of an experienced tester. I’m sure I am missing some. The more experience I get testing, the longer this list will become.