In Agile, developers, testers and business analysts build a working product through a series of short iterations. At the end of each iteration, the product owner accepts the software as working or not, and the team moves forward to the next iteration. This kind of acceptance is not the same as the User Acceptance Testing (UAT) traditionally done at the end of the development effort (Waterfall) by business users or end users who have the knowledge of the product to check it’s functionality and ensure that it can support them in their daily tasks on the job.

UAT is still seen by most as a critical step in assuring quality software and that it meets the needs of end users, however, UAT does not fit neatly into Agile. For those implementing Agile, this often represents a missing link that causes significant pain. In this blog, we discuss how to solve this problem.

Testing in Agile – Where is it?

The Agile Manifesto is a set of ideals or principles. And as with all ideals, they can be difficult to implement in a real life situation. Let’s first examine the Manifesto and what it says and doesn’t say with respect to testing and UAT.

  • Individuals and interactions over processes and tools
  • Working software over comprehensive documentation
  • Customer collaboration over contract negotiation
  • Responding to change over following a plan

As you can see, the Agile Manifesto contains no mention of user acceptance testing (UAT) or testing of any kind. However, the overarching objective of ‘working software’ implies testing of some sort, but what kind of testing and by whom?

The Need for User Acceptance Testing (UAT)

When businesses define their requirements, it is assumed that developers and QA will design and implement software to precisely meet all defined business requirements. Using a Waterfall methodology, all steps are implemented through a sequential and linear fashion. In implementing Agile, complexities combined with quick iterations often limit the level of detail provided and some business requirements may be unintentionally left out, especially where the cliché ‘working software over documentation’ appears to have taken hold of the software industry.

It’s easy to see why testing and UAT gets lost or overlooked. There’s just not enough time and emphasis. Figure 1 below illustrates stakeholders and their lines of communication. As seen in the Figure, End User’s are not clear to begin with in their needs, but when they communicate them, they gain some clarity. However, at each step, their needs and hence the software’s capabilities becomes less clear and uncertainty increases, as they are communicated along the way. Uncertainty increases with time and at each ‘filter’, while intention has less clarity and often grows in scope (illustrated by lighter gray and wider arrows over time). Each role introduces some uncertainty in their thoughts, definitions and outputs. The accumulation of “uncertainty” manifests in defects. For Agile, the question is where should UAT take place? And how can it be inserted or fit into the process without an extra dedicated iteration?

Figure 1. The Need for User Acceptance Testing – Uncertainty Means Defects

UAT Challenges Fitting into Agile

Let’s examine the objectives of UAT and see where we can fit it into Agile. Some classic challenges with UAT as implemented in a traditional Waterfall project that still exist for Agile include:

  • End users don’t have time to thoroughly test as they are pulled from their normal jobs into a testing task.
  • UAT occurs in an unstructured way by simply allowing users to “play” with the system.
  • UAT should be done by the User and from their perspective. Having tests executed from the development perspective won’t assure that this goal is met.
  • Sometimes UAT can take the form of ‘regression testing’, but this not quite the same, as regression testing is still mostly testing from the development organization’s point of view.

In an Agile environment iterations move quickly forward until the final software release. Unless there is a sprint or iteration planned as UAT, it just won’t happen.

Implementing UAT for Agile

To tackle these issues and challenges, we’ve developed an approach for implementing UAT through methods that can be modified to each organization’s flavor of Agile. Rather than force UAT into Agile, let’s examine UAT objectives in order to better understand and define what UAT is. Essentially, UAT should identify gaps between how an application works (as coded) versus how it was and is expected to perform by users in a business environment while executing their required tasks. This means:

  • All desired business requirements have been met. What is and was envisioned by those creating business needs (see Figure 1) is accomplished.
  • Software will be used effectively once deployed, such that from a user’s perspective, the needs, requirements, and associated business workflows have been successfully implemented.
  • Risks of production failures have been minimized.

Besides reducing the risk of defects, the common thread of UAT is satisfying requirements. But what are requirements from an Agile context? In Waterfall, both requirements and testing were phases upon themselves. With Agile, requirements are now part of each iteration in the form of user stories and acceptance criteria.

To successfully achieve the goals of UAT, User Acceptance Testing must be emphasized such that requirements become User Stories. User Stories in turn, should be coupled with acceptance criteria and acceptance tests that are then integrated within each sprint or iteration as shown in Figure 2.

Figure 2. User Acceptance Testing Contents

 

To implement UAT in Agile, we must consider:

  • Where is the User in Agile and how can we get their viewpoint?
  • Who is deciding to accept or not (a part of UAT)?
  • What are they accepting?
  • Who are they accepting for?
  • What do users want out of UAT?
  • Will a decision be made based on UAT and if so, what? What will be the outcome?

For UAT to be successful in Agile, we need ‘good’ User Stories that answer these questions. But what is a ‘good’ User Story?

User Stories in Agile – The Basis of UAT

While there are many formats and examples of what User Stories should or should not be, their purpose is to add certainty to what software can and cannot do. As a general guideline, User Stories address the following:

  • Describe an objective that a user should be able to achieve, or a feature that a user should be able to utilize, when using a software application.
  • In simple language such that a customer or end user would know without ambiguity what the expected outcome is.
  • May reference other User Stories or design documents to provide details, but should not repeat.

User Stories must be accompanied by Acceptance Criteria. Acceptance Criteria should:

  • State intent, but not a solution and should be independent of the implementation and platform.
  • Be comprised of a set of statements, each with a clear pass/fail result, that specify both functional (e.g., minimal marketable functionality) and non-functional (e.g., minimal quality) requirements applicable at the current stage of the project.
  • Represent “conditions of satisfaction” with no partial acceptance: Yes or No, Met or Not Met, True or False.
  • Define the boundaries and parameters of a User Story (i.e. feature) and determine when a story is completed and working as expected.
  • Be easily translated into one or more test cases.

Acceptance Criteria need to be accompanied by Acceptance Tests. These tests are specifically tied to the criteria and if the criteria are written unambiguously with clear pass/fail results, the acceptance tests are merely an implementation of the criteria particular to the related story. Acceptance Tests should be reusable and automated whenever possible.

Implementing UAT in Agile

In Agile, traditional UAT has been swept aside. However, that doesn’t mean that we ignore it or can’t accomplish the objectives of UAT within an Agile framework. Rather, we need to consciously and purposely emphasize the many aspects of User Stories as we’ve described here in this article, namely:

Discipline: User Stories must be groomed on a regular basis. This means revisiting them and adding detail at the right time. Therefore, User Stories should be done for the current and next sprint, but not much farther out in advance. As new knowledge is gained with each iteration, expect that existing User Stories may need further modification.

User participation: User Stories should get user participation and input. We put so much effort into defects via defect triages, so why not prevent the defects in the first place with User Story triages. We recommend using the same triage where the product owner or business analyst, developer, and tester meet to discuss each user story and whether or not there is sufficient detail and clarity.

Communication: Because of the Agile process, everything is moving fast, not only in priority but also definition. User Stories need to be communicated up, down and across with a well-defined communications plan. Setting up the proper communication channels and again, applying discipline is the answer, but that doesn’t mean we aren’t Agile.

By putting proper attention to detail, at the right time, by the right people and making sure that the entire team is cognizant of their meaning and intention, User Stories can form the basis of UAT that is baked into Agile without having a separate UAT phase.

If you have additional questions about User Acceptance Testing, XBOSoft has a team of experts ready to assist. Contact us today!