ISO 29119 – Useful or Not?

“Can’t We All Just Get Along?”

ISO 29119 overview

Is ISO 29119, the new software testing standard useful or not? I’ve been reading a lot lately about the latest ISO 29119 Standard which has recently been released (some sections final and some in draft form) and has received abundant ‘test coverage’ from certain members of the software testing community. There is even a petition of sorts to ‘stop’ it.

Note I say “software testing community”, because I don’t believe there has been any negative feedback from the software engineering or software development community, although testing is all part of the process.

It seems the ‘context’ driven testing community feels that the Standard violates some rules or takes away somehow from what they are doing?

It doesn’t at all. The frightening word perhaps is ‘standard’. This word has connotations of ‘final say’, but as with any ISO standard, the ‘Standard’ is intended as a starting point, guideline, or framework to get our hands around. Usually standards are born from nebulous concepts that we need to try to understand better, thus the ISO 29119 standard was born.

With all the negative feedback on the standard, I think those folks need to just relax a bit. Life is too short to get angry about ISO 29119! When people hold a strong opinion, sometimes it is difficult to listen to–and hear–the opinions of others who have different viewpoints.  For those that oppose the Standard, it certainly takes nothing away from your efforts nor does it detract from your point of view, but perhaps just gives a starting point to add context and customization. In fact, all ISO standards explicitly state that they need to be tailored to the situation and organization. No where does it say, ‘This is the rule’.

We are in the ‘Age of Context’ and to those that promote and practice Context Driven Testing, I think that adaptive thinking is very applicable to today’s world. We definitely don’t want anyone applying the Standard blindly or in the wrong way or context. But it could be a foundation or starting point for someone to start up a testing effort, or perhaps list out things so that they don’t forget. Sometimes people learning from scratch also need a guide or reference.

In the future, we will all need to be more #ContextResponsive and #ContextAware, not only in our software testing, but in our engineering and design and life in general. Sensors all around us are collecting more and more data that is now easily stored (cloud) and accessed (mobile) from anywhere. Harvesting that information and intelligently transforming it into anticipatory and predictive actions and services can truly enhance our lives. But that doesn’t mean that a standard, or framework takes away from any of that.

13 Comments

  1. breakingembedded October 2, 2014 at 8:26 am - Reply

    This is a good pragmatic view point. From ISO definitions of a standard we have:

    3.2 standard
    document, established by consensus and approved by a recognized body, that provides, for common and repeated use, rules, guidelines or characteristics for activities or their results, aimed at the achievement of the optimum degree of order in a given context
    NOTE Standards should be based on the consolidated results of science, technology and experience, and aimed at the promotion of optimum community benefits.

    ISO/IEC Guide 2:2004, Standardization and related activities — General vocabulary

    This does not say the standard is “a best practice”, “Universally approved by a majority of the whole test world”, or “free”. All of these have been made against ISO 29119. The definition does say “optimum community benefits” and “established by consensus and approved by a recognized body”. In the case of ISO/IEEE/IEC 29119, the community and consensus body are made up of organization and people that are members of these bodies and not the whole world of software testers. Users inside and outside of these organization can decide to use (hopefully with thoughtful tailoring) or not use the standard as they see fit. There are testing organizations and people who wanted ISO2 9119, reviewed it, voted on it, and reached “consensus”. Consensus does not mean everyone totally agreed or that any standard is perfect, but this is why IEEE and ISO have maintenance and upgrade processes.

    Pragmatic testers should be aware of ISO 29119 as they should be aware of things such as context-driven testing, math based test techniques. test automation, model-based testing, experience skill-based testing, and requirements verification-checking. There is no universal best and there are things which over time will need to change in ISO 29119, so the current debate about what is “at issue” in ISO 29119 will be used to evolve the standard for those organizations wishing to use it.

    • Philip Lew October 2, 2014 at 8:34 am - Reply

      Yes, we think that pragmatism is the key word here. No organization is forced to use the standard as well, nor is there some certification associated with it. It is merely a guideline for the community. Thanks for contributing.

      • Laurent Bossavit October 2, 2014 at 11:31 pm - Reply

        “nor is there some certification associated with it.”

        That statement is incorrect. Anne-Mette Hass of WG26 has stated: “The ISO 29119 is based on the ISTQB syllabi, and, as far as I understand, it is the intention that the ISO 29119 testing process and testing documentation definitions will be adopted by ISTQB over time.”

        ISTQB is a certification. It is a certification “associated with” ISO 29119 in the reckoning of one of its main authors.

        • Philip Lew October 3, 2014 at 7:30 am - Reply

          True, perhaps ISTQB may use some of the standard as part of its basis. But, there is not direct certification associated with ISO 29119. For instance, there is no ISO 9001 certification type audit. The standard, as other comments have alluded to, is mostly for those who want to use it, or not. Its up to the organization. If it ever gets to the point where companies have to be certified as compliant with ISO 29119 then that certainly would be the wrong direction, but don’t think that will happen.

          • Laurent Bossavit October 3, 2014 at 8:36 am

            In what fundamental sense is “ISO 9001 certification type audit” different from the situation with ISTQB and 29119? ISO does not directly certify anyone:

            “At ISO, we develop International Standards, such as ISO 9001 and ISO 14001, but we are not involved in their certification, and do not issue certificates. This is performed by external certification bodies, thus a company or organization cannot be certified by ISO.”

            The only difference between 29119 and other ISO standards that do have certification schemes associated with them is that 29119 is relatively new and there hasn’t been time to ramp up that commercial activity.

            However, Anne-Mette Hass’ statements regarding ISTQB and ISO 29119 certainly point to the possibility that this will in fact take place in the future. There is ample factual grounds for believing that this will happen, and no factual grounds for hoping that it will not happen.

  2. dsynadinos October 3, 2014 at 8:45 am - Reply

    “No where does [the standard] say, ‘This is the rule’.”
    Look up and compare the definition of “standard” and “rule”. Outside of law, they both speak to explicit, required, agreed-upon prescriptions and expectations. For all intents and purposes, a standard IS a rule.

    “It seems the ‘context’ driven testing community feels that the Standard violates some rules or takes away somehow from what they are doing?”
    To me, this is an inaccurate representation of the context driven testing community’s objections to 29119. Even if you don’t agree with the objections, you should at least restate them accurately.

    To me, it seems that much (but not all) of the 29119 debate could be summed up as:
    Supporters say “do it ONLY according to the explicit, required, agreed-upon way described in the standard”
    Detractors say “do it ANY way that makes sense (which may be none, some, or all of the way described in the standard)”

    The problem is that not everyone is as thoughtful as you and I. I’ve written the idea below elsewhere and will paraphrase it here:

    I suspect that for each thoughtful 29119 supporter and detractor that takes time to read, write, think and debate about 29119, there are many thousands of uninformed testers, managers, executives and companies that do not. Unfortunately, these folks are also often in charge and looking for a quick and easy solution. These are the folks that will rarely (as one of your commenters said) “thoughtfully tailor” a standard as they see fit. These folks often command their reports (like me) to “blindly follow the standard” without regarding the context. And these are the folks that I fear.

    And so, I can’t simply “Get Along”.

  3. dsynadinos October 3, 2014 at 10:32 am - Reply

    (Looks like the form ate my last comment submission. I’ll try again…)

    “No where does [the standard] say, ‘This is the rule’.”
    Look up and compare the definition of “standard” and “rule”. Outside of law, they both speak to explicit, required, agreed-upon prescriptions and expectations. For all intents and purposes, a standard IS a rule.

    “It seems the ‘context’ driven testing community feels that the Standard violates some rules or takes away somehow from what they are doing?”
    To me, this is an inaccurate representation of the context driven testing community’s objections to 29119. Even if you don’t agree with the objections, you should at least restate them accurately.

    To me, it seems that much (but not all) of the 29119 debate could be summed up as:
    Supporters say “do it ONLY according to the explicit, required, agreed-upon way described in the standard”
    Detractors say “do it ANY way that makes sense (which may be none, some, or all of the way described in the standard)”

    The problem is that not everyone is as thoughtful as you and I. I’ve written the idea below elsewhere and will paraphrase it here:

    I suspect that for each thoughtful 29119 supporter and detractor that takes time to read, write, think and debate about 29119, there are many thousands of uninformed testers, managers, executives and companies that do not. Unfortunately, these folks are also often in charge and looking for a quick and easy solution. These are the folks that will rarely (as one of your commenters said) “thoughtfully tailor” a standard as they see fit. These folks often command their reports (like me) to “blindly follow the standard” without regarding the context. And these are the folks that I fear.

    And so, I can’t simply “Get Along”.

    • Philip Lew October 3, 2014 at 10:50 am - Reply

      Well so much to think about here. Firstly I disagree that a standard is a rule but that’s ok.
      I perhaps am a little naive to think there are really any real supporters as you define them. Who in the world would blindly use a standard? And lastly if you boss is this type who’d force a standard upon your work without thinking of where and when it should be applied then maybe that’s not the right place for you?

  4. dsynadinos October 3, 2014 at 10:33 am - Reply

    (And, trying for a 3rd time…)

    “No where does [the standard] say, ‘This is the rule’.”
    Look up and compare the definition of “standard” and “rule”. Outside of law, they both speak to explicit, required, agreed-upon prescriptions and expectations. For all intents and purposes, a standard IS a rule.

    “It seems the ‘context’ driven testing community feels that the Standard violates some rules or takes away somehow from what they are doing?”
    To me, this is an inaccurate representation of the context driven testing community’s objections to 29119. Even if you don’t agree with the objections, you should at least restate them accurately.

    To me, it seems that much (but not all) of the 29119 debate could be summed up as:
    Supporters say “do it ONLY according to the explicit, required, agreed-upon way described in the standard”
    Detractors say “do it ANY way that makes sense (which may be none, some, or all of the way described in the standard)”

    The problem is that not everyone is as thoughtful as you and I. I’ve written the idea below elsewhere and will paraphrase it here:

    I suspect that for each thoughtful 29119 supporter and detractor that takes time to read, write, think and debate about 29119, there are many thousands of uninformed testers, managers, executives and companies that do not. Unfortunately, these folks are also often in charge and looking for a quick and easy solution. These are the folks that will rarely (as one of your commenters said) “thoughtfully tailor” a standard as they see fit. These folks often command their reports (like me) to “blindly follow the standard” without regarding the context. And these are the folks that I fear.

    And so, I can’t simply “Get Along”.

  5. breakingembedded October 3, 2014 at 12:22 pm - Reply

    The “blindly follow the standard” issue exists and has always existed, and will always exist. There are historic test books of knowledge, training, ideals, and standards where the unthinking (maybe lazy) will take the easy path of “blindly follow the standard”. Doing away with the documents and standards will not stop people from doing this. Testers can vote with their feet (there are jobs and companies that do think) or do what I did which was to think and educate the unthinking leaders I worked for (I only had to walk away from one job out of 35 years). The 29119 standard allows tailoring and that can be pointed out in the education process. One of my big jobs as a tester was to provide information (about bugs, performance, quality, etc). Providing information about how to correctly use a standard was one thing I did to my leaders. It took time, effort, thinking, and politics. It was how I become recognized as a leader and not just a tester. It is not easy.

    As to the issue of certification and what Anne-Mette Hass has said. She was voicing her opinion. She can not speak for all of ISO or ISQTB, so some care must be taken. However, she is pointing out what will likely be a logical progression to consistency between books of knowledge (the ISQTB documents) and standards (ISO 29119). They are different organizations. The progression will take time.

    ISTQB is used in certification of individual testers to pieces of knowledge. For me, ISTQB does not indicate real tester skill or ability. Knowledge and skill are different things. One can have knowledge (like coming out of college) and minimal skill. Testing takes skill.

    ISO 29119 is to be used by companies/organization in conducting business. There is a plan to have a model which is based on ISO 29119 to accredit organizations. As ISO states use of such a model for ISO29119 and issuing compliance would be done by a third party, not ISO or IEEE. This is how concepts like the CMMi work and organization may seek such compliance for business reasons. There may be training organization which offer some kind of “cert” or even college degree for testers to claim knowledge of ISO29119, but again this is a third party. To some extent everyone working in software test and software test industry are “rent seekers” as some critics have pointed out (Yes I know there is open source software and other things produced for free in our industry). So Third parties issuing certifications and compliance statements will happen. But then there people and companies which claim to follow context-driven. If someone is really following context-driven and/or doing test practice, I do not see where making claims are needed. The proof will be in software that works, makes people happy, and inspires passion.

    BTW, under disclosure, the above are my opinions. I too am a member of WG26, past member of ASTQB, member of IEEE, member of AST and many others. I am retired (mostly) and do what I do in software testing because I want to see the industry improve.

  6. Michael Bolton October 3, 2014 at 12:35 pm - Reply

    I encourage you to relax, too; that way, when the certificationists show up for another round of pocket-picking, you’ll barely even feel it.

  7. Thomas Ponnet November 3, 2014 at 6:50 am - Reply

    The ISO standard is not a rulebook, true. It becomes a rule when in a contract contains “…will be tested according to ISO 29119…” At that point there is no leeway to test in a way that is best for the project. Many professionals myself included can easily see that customers given the choice between someone promising them to do their project work either “to an international agreed standard” or “what’s best in context of the project” to the former.

    My problems with this standard is that it’s a marketing tool on one hand (give the project to us, we’ll test it according to ISO 219119) and a working horse for selling expensive ISTQB certifications. I haven’t figured out what the cart and what the horse is, only time will tell.

    • Philip Lew November 4, 2014 at 10:27 am - Reply

      I don’t see ISTQB based on any ISO standards thus far, so I am not sure if they would include this as a part of the ISTQB certifications.
      “test according to ISO 29119” perhaps is a fear and a valid one, because that would be very stupid to say. Some parts of ISO 29119 could be applied well to one project and have zero value on another. I’d hope that people could intelligently use these standard as a guideline, but you are right, if they say “USE ISO 29119” in some government procurement that would be deadly. Maybe that’s how the healthcare.gov site got messed up!

Leave a Reply

Show Buttons
Hide Buttons