In this post, we’ll cover Habit #3: Sustain an Improvement Frame of Mind. At XBOSoft, we use these 7 “habits” to guide our everyday work. They’re all important, but I think Habit #3 is all encompassing of the others.The key word here is “maintain.” One of the key tenets of agile is sustainability. If you can’t sustain an effort with a constant pace, then you’ll soon fall behind and lose the race. And what better way to understand your pace than metrics?
I've been to numerous talks on software measurement and metrics, and it seems inevitable that the instructor quotes this guy named "Lord Kelvin." I finally decided to figure out who this guy is and see how he is related to software measurement. I found out that Lord Kelvin was a physicist named William Thomson. According to the Wikipedia, "He was ennobled in 1892 for his achievements in thermodynamics, and of his opposition to Irish Home Rule and became Baron Kelvin of Largs in the County of Ayr. He was the first British scientist to be elevated to the House of Lords, and Kelvin refers to the River Kelvin that flows close to his laboratory at the University of Glasgow." Being in the House of Lords + Baron Kelvin = Lord Kelvin. With his scientific research in the laws of thermodynamics, absolute temperatures are stated in units of kelvin in his honor. Although the existence of a lower limit to temperature (absolute zero) was known prior to his work, Lord Kelvin is widely known for determining its correct value as approximately −273.15 degrees Celsius (-457 Fahrenheit).
So why is this guy in the teachings of software measurement and metrics? Well, he did make a lot of very relatable quotes in scientific journals and as a professor.
GQM stands for the Goal Question Metrics approach, promoted by Victor Basili from the University of Maryland. He is often given credit for the approach, after supervising a Ph.D. thesis by Dr. David M. Weiss. Weiss' work was inspired by the work of Albert Endres at IBM Germany, so I suppose that Endres, Weiss and Basili were all co-inventors of the approach and possibly others. This approach has been widely used in different industries, but often applied to software measurement. In a recent talk at the QAI Quest Software Quality Conference in Chicago, I applied GQM to Agile, calling it Agile Goal [...]
I've been to many sessions on software testing metrics where the instructor will discuss the software testing Hawthorne Effect. Often cited are lighting experiments done in the 30's where the light in a manufacturing facility was increased and decreased and its positive effect on factory workers' productivity. When applying the results of these experiments to software testing, most will then discuss testing metrics such as test cases written or defects found, and the unexpected consequences or changes in behavior that can result from using such metrics. But I think we are missing the importance and significance of the Hawthorne Effect. First, the Hawthorne Effect was based on several experiments not only in lighting but also, in many other factors such as break times, food, and payment schemes. Secondly, the interpretations of the Hawthorne experiments vary, and many researchers have derived different conclusions. Some of their conclusions, I hereby summarize as the Hawthorne Lessons:
GQM (Goal Question Metric) is a methodology for aligning goals with metrics for a business process. Although used mostly in software development environments, GQM is also used in many business management environments. But the tough part is asking the right questions. With our upcoming half-day tutorial titled "Agile Metrics – It’s Not About Velocity" at QUEST 2016 in Chicago on April 19, I sat down with Phil Lew, our CEO, to discuss some key GQM questions that come to mind when using the paradigm in light of Agile. Here's just a sampling of some of the questions.
Relating Metrics Back to Our Agile Objectives
When designing metrics to measure your agile process, it's important to recognize that not all metrics are fit for all organizations. It wouldn’t make sense to measure blood sugar everyday for a patient not prone to diabetes who shows no symptoms, so it wouldn’t make sense to buy any of the special equipment to do so nor spend the time to do it. The point is that to determine the optimal metrics for your organization, you need to examine your current processes and determine where your problem areas are as well as your goals. If we are an organization that uses an agile development process, let’s start by looking at two key agile objectives of quality and velocity:
Our most recent webinar, Are You Making These 7 'Testing Metric" Mistakes?, with ARGO Data's Mark Bentsen produced a question from a participant regarding unit test coverage that we didn't have time to answer live. We asked Mark to share his response and here's how he answered.
Managing agile teams is often tied to metrics and measurements. I was in a recent talk on metrics and many were discussing agile metrics. One of the key points repeatedly raised was that you should never have individual metrics, only team metrics. So as I sit here writing evaluations, how should I be evaluating my team members if I have no individual metrics?
Some folks might interpret "Working software over comprehensive documentation" to mean that there should be no agile defects written up. Shouldn't all defects be fixed for "working" software within an iteration? There is optimal collaboration within the sprint and thereby there is no need for an actual defect to be written up. Right? The only problem with this scenario is that is ideal, but rarely happens in reality.
Just mention software testing metrics and there is ample discussion on Defect Removal Efficiency (DRE), so let's examine it closer.
Back in the days of waterfall, if we found 100 defects during the testing phase (which we fixed pre-production) and then later, say within 90 days after software release (in production), found five defects, then the DRE would be what we were able to remove (fix) versus what was left and found by users:
100/(100+5) = 95.2%
This 95% has been referred by Capers Jones, a well known software measurement colleague, as being a good number to shoot for.
After making the rounds doing talks at both Mobile Dev + Test in San Diego and Better Software West in Las Vegas, I've had a chance to reflect on some of the comments and discussion during my sessions and there is one resounding theme. Value.
We can think of defining software quality in many terms such as defects, requirements, code quality, etc. But when you think about it, it all has to do with the value for the end user. Value is not usability, and value is not defects found and fixed.
I recently came upon an article and corresponding infographic on software sizing, the basic gist of the webinar and article was that properly estimating software size is important? Why? Well obviously, if you estimate incorrectly you have some problems:
- Estimate too high, and you allocate too many resources and spend money you wouldn't do otherwise. I don't think too many organizations worry too much about this situation.
- Estimate too low, which is usually the case, and you end up missing deadlines. But let's think about what this means at a deeper level:
We're back again with outstanding questions from our Agile Metrics Webinar. This question relates to Defect Aging.
Q: Slide 63 - Is this also called Defect Age?
A: Yes, what we term as Time to Fix is equivalent to Defect Age, but a little different. With Defect Age, you could measure all the defects that you have fixed and
Finally the last question from our Agile Metrics Webinar. This question pertains to Rework From Defects. Q: Is Defect rework allocation = similar to root cause type mark up in defects after fix? A: Yes! We use a different terminology because we want to focus on Rework From Defects. In other words, what time are we wasting due to defects. So by classifying where the defects are from, we can determine where the rework comes from, and where we should put our focus on getting more efficient and thus FASTER, increasing velocity. If our rework comes from testing, we may want to look at
Yet another question during our Agile Webinar. This one on Total Unexpected Work. Q: Question on Total Unexpected Work: Is that Metric for total work done in # of hours, and what is work? A: Work can be classified as any tasks that you do. This could be understanding requirements, design, defect fixing, working on new features, and meetings. Each of these work types should be classified into buckets or categories that make sense for your organization. You don't want to be too detailed because then data doesn't accumulate into meaningful amounts. Regarding measurement units, work can be measured in
During our Agile Webinar on Dec 11, 2014, we had so many questions we couldn't get to, and still answering them. Here's one on Agile Unplanned Work and Scope Creep. Q: Adding scope means analyzing the risk to the business or to the customer's usage A: You are right if you add stuff to do, and don’t change other things. But if you add features or stories within a sprint and don’t take out things,
In our webinar on Dec 11, 2014 on Agile Metrics, the questions were overwhelming. We are getting to them one by one. Here's one on the metric of agile sustainability as it relates to working overtime. Q: Remember you don't pay full time employees OT since they are on salary fror the most part. A: Remember, we are talking about agile sustainability. A sustainable pace is what is important, not whether or not we are getting paid for overtime or not. Perhaps getting paid for overtime makes it a little less painful, but
Agile Metrics Webinar Question: What does Technical Debt include? Answer: The debt can be thought of as work that needs to be done before a particular job can be considered completeor “proper.” If the debt is not repaid, then it will keep on accumulating interest, making it hard to implement changes later on. Unaddressed technical debt increases software entropy and is usually as a result of poor system design. Think of all the things that you say “we’ll do that later”. That represents technical debt.
In our recent agile testing webinar, we had an outstanding question on Defect Removal Effectiveness (DRE). Query1: For DRE calculation, I believe below is the correct measure. Can you please double confirm? DRE = Defects found pre release/DFP+Defects found in prerelease Yes, your calculation is correct. Ours is actually the inverse and would be called Defect Escape Rate (wanting to keep low) instead of Defect Removal Effectiveness. Let’s do an example just to illustrate.
Going through notes for the metrics one day workshop at PNSQC entitled "Software Quality Metrics to Go", we applied Basili's GQM (Goal Question Metric) paradigm and methodology to software quality. In particular, we called it Agile GQM, where we came up with 2 primary goals for agile, quality and speed (velocity) and input that to the GQM paradigm. Given the goal of quality, we also focused on the end user point of view, and came up with some interesting questions:
Sometimes there a a need to party, and when we're able to increase Defect Removal Efficiency for our clients, we party up. Metrics are a key part of measuring our service provided to our clients, so we track it carefully. Not only do we need to measure the DRE % in aggregate, but also by tracking pre-production defect issue types versus post-production issue types, we can determine where we need to focus more effort, and possibly beef up our knowledge. For instance, if most post-production defects are found on certain platforms then perhaps we can look closer at deployment configurations or particular platforms. On the other hand, if certain functional areas have more defects in production, then we know somehow these escaped and that we need either more training in these areas or more focus.
Of course we don’t want to waste our clients’ time, and that's why we measuring test effectiveness is critical. For any defect we report, we want it to be a real defect, so we calculate these metrics: Defect finding capability: real defects found / total defects found Defect rejection rate: defects rejected by client / total defects found This can measure how well the testers know the application. What is important here is to analyze why defects are rejected or invalid. Some possible reasons: Could not be reproduced or repeated Could not be understood, not enough detail Not a defect – [...]
The goal of agile is be faster and produce better software. To do this, we need to reduce and minimize rework. One key metric for measuring rework is:
- Fix Effort Ratio - Time to fix bugs/Total effort expended (hours)
As a dedicated testing company, we want to provide as much Software Testing Value as possible by focusing on testing our clients’ software and ensuring that it doesn’t get into the field with post-production defects. Sometimes our client finds defects too as we are not the only QA engineers on the team. Of course we want to minimize that or they don’t need us! To track this, we have a few metrics:
- Defects found by client / total defects
- Defects found by XBOSoft / total defects
For Agile projects, everyone is concerned about velocity, quality and working software. So it makes sense for agile test metrics to carefully track anything that would impede those objectives. Test metrics can be used to help determine whether a project is on track. Metrics such as number of tests run and passed, code coverage and defect metrics can give only part of the picture. Like any data, they need interpretation in context and need to be examined as they relate to the objectives of the team and project.
As I prepare for my talk in Kuala Lumpur at Softec 2014, I started thinking about our own projects at XBOSoft and the software defect metrics that we use internally to see how we are doing for our clients. There are the normal ones such as defect escape or leakage rates, and defect fix time, technical debt reduction - refactoring. But from a 'pure qa' point of view and in particular XBOSoft, we want to reduce the work for our clients while improving the quality of their software. Some of the key metrics we track include:
During the ASTQB 3 hour workshop, Software Testing Metrics Are For Action, one of the top questions was the use of DRE, or Defect Removal Efficiency. I was surprised to find that out of the 40 participants only 5 used this metric (12%). This number was similar to a workshop I was in last year at PNSQC where the material presented indicated that even less than 12% considered this metric essential in measuring as part of a metrics program. The class was lively and spirited. We divided the class into 3 modules:
As mentioned in a previous post on End User Perspective of Quality, quality means much more than conformance to requirements and the end user perspective on quality, sometimes called the User Experience (UX), is becoming more and more important. Measuring UX is a tough task but its important an any attempt to quantify end user quality and quality in general. Many often give up thinking that there is just too much subjectivity. I'd think measuring UX can be done from another viewpoint. UX depends on many factors, not just the software interface and the 'wow factor' of the interface, but the entire experience in interacting with the company. An end user's overall UX is also therefore influenced when they call your help desk or technical support. Some folks refer this as the front line in dealing with the customer/end user. Current metrics in measuring service desk performance are rarely tied to software quality
XBOSoft kicked off its Knowledge Bites series on March 18. This series is designed to take a deep dive on specific topics quarterly. This quarter, the 3-part series partnered with CEO and Founder of SQE Labs, Shilpa Dadhwal. The focus of these session was how software quality and the right metrics can drive business value. She identified the framework to understand software quality and software quality metrics. Specifically, she discussed who owns different measures and how to measure the attributes accurately and how to tie it all together so that organization can leverage all the metrics effectively. All easier said [...]
This is the last batch of questions from the webinar as a pre-cursor to the session I'll be doing on software quality measurement for the Quest 2014 conference. These questions focused on something really relevant in my research which is a "software quality scorecard"
Many agile metrics are related to velocity since agile is supposed to deliver product faster. That makes sense, you want to measure and manage what it is you want to achieve, speed. Perhaps you think less documentation, no need for a test plan, etc... leads to speed! However, velocity should only be one of several agile software quality metrics. And just thinking and focusing on velocity can have detrimental effects. After all, if you run full speed constantly, the body breaks down. Many forget that one of the keywords in the agile manifesto is 'sustainable'.
In an attempt to get to all the questions from the webinar sponsored by QAI on software quality and metrics in preparation for the Quest 2014 conference, I'm attacking them a few at a time. How do you balance (or minimize) the effort involved in collecting and summarizing the results of the metrics to impact to the project work being done?
Still trying to get to all the questions from the webinar as a pre-cursor to the session I'll be doing on software quality measurement for the Quest 2014 conference, I'm attacking them a few at a time. We also recorded the webinar, so you may want to view it here. What is the general improvement (cost %, time % of overall budget/duration of project) seen by implementing a measurements & metrics framework across the industry?
In a recent article I published at Software Test Professionals, one comment I received was that it was regarding Table 1 indicating that different people are likely to be interested in different things with regard to software quality and test metrics, and that many stakeholders were not included in the table. What other stakeholders would you include? Another comment regarding my list of "why metrics/measures fail” stating that since it was a listing of common reasons for failure, the implication is that there are other reasons. Certainly there are many reasons for failure of metrics programs
In the recent webinar with QAI on software test metrics, the audience has several questions that we were not able to answer during the webinar. Here are the first few questions and related answers. Any suggestions for measuring resource allocation?
In the recent webinar "Software Quality Metrics - Do's and Don'ts", with ASTQB, we ran a few polls to get an idea of how software quality professionals are really using, OR NOT Utilizing Software Quality Metrics. It turns out that:
In today's webinar, Shaun Bradshaw with Zenergy Technologies, discussed how to use the 'S-Curve' in managing testing projects. First he discussed metrics in general and where they should can should not be used. In particular, he discussed context and how there are always interpretations and assumptions that go with metrics. These interpretations and assumptions are key in using metrics and are one of the most prevalent reasons that metrics fail.
Then he discussed...
Some consider test metrics and in particular, managing with metrics, a thorn in software development and testing, but when used properly, they provide valuable insights into what occurs during projects and what strategic and tactical adjustments must be made on a daily basis. Find out how a small set of test metrics were used to successfully manage a major test acceptance effort. This was done at the conclusion of a two and half year ERP implementation. Bradshaw brings to attention specific use of the S curve analytics. He provides an interesting history on how the "S-curve" was first discovered. Obviously, [...]
Here we go again...even more valuable information from Jay Philips and Mike Lyles, two of the panelists from December's lively discussion about Software Metrics entitled, The Good,The Bad and the Metrics. Test Metrics Love 'em or hate 'em everyone has an opinion about them... some even have questions about 'em.... Q. What are the purpose of metrics and how can they be done 'well'?
For those of you who were wrapped up in holiday madness and may have missed the December webinar The Good, the Bad and the Metrics featuring Jay Philips, Mike Lyles and Rex Black- fear not we've compiled (OK we nagged a little) the answers from two of our very busy but also gracious panelists from this very interesting discussion. For those of you who joined us for the live presentation thank you for your patience. Here is Part 1 of the Questions/Comments that attendees sent in that were not previously addressed. This is so much valuable information to share that it will be split into 2 blog posts.
San Francisco,CA (PRWEB) December 16, 2013
The Good, the Bad, and the Metrics, webinar focused on when to use and not to use software testing metrics.On December 17, XBOSoft hosted another complimentary software quality webinar; this time on software testing metrics. Are metrics worth the effort? Speaker Jay Philips agrees,"Bad metrics are worse than no metrics …" Where and how does your organization use metrics and measurement if at all? Are your current metrics clear or perhaps misinterpreted?
Yesterday, I finished my presentation at Starwest on Software Quality Metrics For Testers. It was a packed room and I was really surprised. I started by discussing the title, why 'quality metrics' instead of 'testing metrics'. I like to use analogies, so I started out by discussing diet and health, something everyone can connect with. With diet it's easy to see that what we eat affects our weight, and our weight in turn impacts our performance and results if we engage in sports. Using that we can compare that to testing, where activities done well early in the development lifecycle such as requirements can influence code quality, and satisfaction of requirements. This in turn, can impact the test results, and in the end, the users.
Agile testing metrics are no different than metrics for other development methodologies, except for timing. Using agile, we want progress to be highly visible and problems (and potential problems) to be known immediately. We can’t spend days calculating metrics when we want to know what’s going on for that day. Collecting data, and then calculating and reporting should be easy and valuable, good ROI. We also need our metrics to be directly connected with goals and questions-answered for our stakeholders. Otherwise, they’ll walk by the whiteboard and won’t care. Since one of the main reasons or objectives of stakeholders with regard to agile is deliver high quality code that works, and can be considered deliverable product, Continuous Integration (CI) is an integral component of agile. So, our agile testing metrics should be somewhat connected to the CI.
Have you implemented a software quality metrics program? If so, then you know its not easy and that metrics programs often fail. Why? Software Quality Metrics - Top Reasons for Failure --Metrics don’t measure the right thing --Metric becomes the goal --Metrics don’t have consistent context so they are unreliable --Measurements used are not consistent --Metrics don’t answer the questions you had to begin with --Metrics have no audience --Measurements are too hard to get --Metrics have no indicators so cannot evaluate
We are happy to announce that March 20 at 10 AM EST we are holding an agile testing webinar with two veterans in the field: Jared Richardson and Philip Lew. Jared and Phil will discuss the changes needed for QA and Testing when working in an agile environment. Agile Testing Topics covered include: - Agile development trends - How to test 'agile' - How to implement scrum - Typical scrum testing bottlenecks and how to solve them - Testing agile requirements - Agile test metrics For Jared Richardson & Philip Lew Jared Richardson Jared is a software consultant. [...]
October 25 at 10 AM EST we are holding a webinar in which we explain how XBOSoft’s Quality Process Assessment (QPA) can help you prevent defects throughout your software products’ entire life cycle, above and beyond development and testing. We show how to examine your software product's lifecycle and how each phase impacts your customers' view of the product's quality. This includes phases such as: - Sales & Marketing-Product/Project Sponsor (intra-Company software) - Product Management (Requirements) - Development - Quality Assurance - Customer Service/Tech Support In this webinar consultants from XBOSoft will share with you their first-hand experience. You'll learn: - [...]
Software quality is an abstract concept. So abstract, that standards bodies develop models to understand it better, i.e. ISO 25010. Quality models help us to crystalize and understand quality and quality requirements, both functional and non-functional, with the goal of evaluating them. The goal of testing is to determine if these requirements are met. During the course of testing, we find defects, or instances where the software does not meet requirements. Hence, in the area of software testing metrics, there has been abundant work in analyzing defects. With respect to analyzing defects, there are many flow charts detailing how defects flow back and forth to QA with changes in state (fixed, open, re-open, etc.). There are also numerous software applications (defect tracking systems and defect management systems) that help us track defects at the different phases of development and after release. However, these activities are rarely connected to metrics in such a way that is easy to analyze. Rather, there many defect metrics often listed out with definitions and calculations
In a nutshell, establishing and implementing a measurement program for software quality needs more than software testing related metrics. Testing metrics (like defect statistics) are detective type metrics which are very late in the product life-cycle, and also only include one part of the organization. What about metrics that could prevent the defects to begin with?
In my last post, I mentioned that metrics could be used for different purposes. One of them was tracking progress. Usually when we track progress, it is related to time, or other unit that indicates a schedule. Most often we use progress metrics to track planned versus actual over time. What we track depends on our role. If we are financial people, then we track money spent. But for software quality assurance, we want to track progress of such things as defects, test cases, man-hours, etc. Basically anything that is related to results or effort spent to get those results. For instance, here are a few: