On my current company’s blog I posted some time ago part one of “writing good tests”. Now it’s time to continue.
Last time I just barely scratched the topic of writing test scenarios. I briefly gave only a few hints like “avoid unnecessary information”, “don’t include implementation details” or “don’t repeat yourself in different scenarios”. This time I’d like to dig deeper and present results of discussions we had in the team.
Note: Like I mentioned last time we’re developing our applications Behaviour Driven Development way using Cucumber. I’m sometimes referring to how scenarios are written for Cucumber in Gherkin format, so please make sure you have at least a rough idea about it.
Purpose
Scenarios serve two purposes:
- feature specification that helps clarify what exactly needs to be done and
- a base for executing automatic acceptance tests.
To fulfil those purposes, they need to be understandable by both business and developers.
Features orientation
Acceptance tests are about features and functionality, scenarios should abstract all implementation details of how they’re implemented. Especially, they shouldn’t mention anything about applications, components or use any other technical terms. Scenarios should be written in a business language using terms from the business domain. The reason is: restructuring or refactoring of the code, moving functionalities between modules or applications have no influence on features themselves, so scenarios shouldn’t change.
Example: something like “when the application is executed” should be replaced with the desired functionality like “when emails are sent”.
Implementation of tests, on the other hand, has to be and is implementation dependent. Implementation of each step should do exactly what the step actually says. In cases where this is not possible, they can be implemented in some other way, but which would still ensure proper behaviour of the system.
Example: for a legacy code it might be difficult to just call some service and the only way to verify the results is to do something else like just running the application. That’s OK (for now) as long as that is the closest place to verify the desired behaviour.
How to write scenarios
The idea is pretty simple. Scenarios should be written in a top-down manner, so they should start with very high-level features and later on add more and more details, define higher level concepts or terms using lower level concepts. The scope of each scenario should be as small as possible, should refer to only one feature or requirement and mention only things important or influencing particular functionality. Each scenario should define exactly one term and can use other terms to do so. If those other terms are not defined yet, each of them will require later on a scenario or dictionary entry. It’s important to make sure, that each scenario contains exactly one definition.
Example: Given a customer And the customer selects an article And the customer enters a desired quantity of that article When the customer adds the article to the cart Then the cart contains the article with the given quantity
This scenario is about adding articles to cart in some e-shop. It defines the term “customer adds the article to the cart” by specifying what needs to happen before and desired outcome. To do that this scenario is using other terms like “customer selects an article” or “cart contains the article with given quantity”. Those terms require scenarios on their own.
Dictionary
As mentioned previously, some terms might be defined in a dictionary rather than in a scenario. A dictionary is just a list of terms and their textual definitions or descriptions (that is they’re not meant for test automation so they don’t need to conform to any rules like Gherkin format, they just clarify things). A decision for each term needs to be made on a case by case basis. By default, terms should have their defining scenarios. But if:
- a term refers to the outside world (for example something defined in another part of the system) or
- it’s a common knowledge (volume = length * width * height)
then the term can be described in the dictionary.
Implementation details abstraction
Like mentioned before, all implementation details need to be abstracted away from scenarios. It’s not part of a business requirement that, for example, something should be saved into the database. If implementation team would decide to use different technical means of achieving the same goal, like persistence, then the test based on the scenario would be “red”, but actually, the business goal would be achieved. This will give the team much more freedom to solve technical problems as they see fit and also change the implementation if necessary, but still, make sure that the features are working.
Of course, sometimes requirements are to save something to the database. Such requirements should not be coming from “business” though, but might from other “technical” teams like operations. For purely technical modules those DB details will actually be THE thing that needs to be verified. In all those cases such technical details can and must be mentioned in scenarios themselves.
Exhaustiveness
Scenarios should be MECE (mutually exclusive, collectively exhaustive): they need to cover all business requirements, including seemingly obvious ones. There should always be test scenarios for business error cases, meaning cases where something wrong is expected to happen and then such scenarios need to specify what to do.
Simplicity
Scenarios should be as simple as possible. Any unnecessary information or data makes tests more difficult to understand and obscures their purpose. Test scenarios serve also as specification, so if something doesn’t influence the feature, don’t mention it.
Completeness
A single term should be defined by exactly one scenario. All exceptional cases should be covered together with “happy path” in one scenario. Use “Examples” clause to explain all the cases there are for a given feature.
Precision
Scenarios should use exactly the same wording and phrases for describing the same conditions or situations. It will reduce confusion like “are we talking about the same thing or is it something different?” and also will make tests implementation easier. Be consistent about names and abbreviations, always write them the same way (upper/lower-case, written together or separately). Scenario summary (“Scenario” / “Scenario Outline” lines) should use full form, steps can use abbreviations.
Other useful hints
Test scenarios should be completely separated from any names or identifiers used in production. So “country 1” is preferred over “Germany”. This would give additional benefit of making sure, that no such name or id was hardcoded inside the code. It would also ensure, that the feature is generic and can be applied to another entity (so no fixed features for, say, country Germany, but a generic feature that can be enabled for Germany).
Similar to previous advice, hardcoded numbers or values should be avoided, because they’re usually obscuring real purpose or condition. So it’s better to replace something like “given operation takes more than 30 seconds” with “given operation times out”. It will make test scenarios independent from configuration parameters. This would also violate DRY principle (don’t repeat yourself).
All test scenarios for one feature should be kept together, in one file if feasible. Otherwise, it’s a single responsibility principle violation, things related to each other should be kept together.
Scenarios should be written in such a way, that they can be read “in English” (or whatever language you’re using), technical forms should be avoided if possible. For example “user adds an article to cart” is better than “user clicks ‘add’ button”.
Scenario summary should be both precise and concise. One line, not so long, should be the target size.
Use passive voice in scenario summaries. So “user is logged in after providing valid credentials” is probably easier to read than something like “when the user provides valid credentials then he/she is logged in”.
Scenario summary should describe the general property, steps can be more concrete. So summary could contain “cheapest option is selected out of all possibilities” because this is exactly how the feature should work, it doesn’t matter how many possibilities are there. In the steps mentioning two options and showing the behaviour is perfectly OK. Steps are there to clarify and give examples, but the summary should be 100% precise.
Summary
Like mentioned in the beginning the hints above are results of many discussions we had in the team. We have found it difficult to write those scenarios, sometimes it was taking a similar amount of time than actually implementing those features. That’s why I’m sharing this, maybe someone else could use it and avoid lengthy (and heated) discussions.
I do not pretend to say, that this is the only way to do it. It works for us™, YMMV. I’d be extremely happy to also get some hints from others, so if you’d like to add something in here or you disagree with what I’ve written above, feel free to leave a comment.
BTW, this post was simultaneously posted on two blogs: Zooplus Tech Blog and my private one. Please feel free to go to the other one and check it out!