Moving beyond the basic ‘it works’ functionality, many of our clients are now asking about taking performance testing to the next level. Where 3-4 seconds used to be ‘good enough’, now users expect 1-2 seconds response time and even less for frequent usage scenarios. As “performance” expectations continue to increase from end users, what can a Tester do in response?
Below, we listed out a three-step pathway to the next level in performance testing.
Step #1: Questions to Ask
In the age of Google where all the answers are somewhere in the ether, the most critical skill is not finding the answer, but asking the right questions. In light of this, we’ve put together a list of performance testing considerations. This list is by no means all-inclusive, but can be considered a good start.
- Who are your users? Break your users down into types based on what they do or their user privileges. Some will be admins, while others may only do one thing and one thing only. In testing or business analyst terms, develop user personas not just for functionality but performance expectations.
- What are your users really doing? All your users won’t be doing the same thing at the same time! Think in their shoes. What are they searching for? What is their job role?
- When are they doing what? Depending on your application, different tasks will be executed at different times. If your software is accounting software, then end of month and end of quarter will have certain activities where many users are accessing similar data.
- How often do they do these things? If your software is teleconferencing software, then maybe Mondays are peak days, and top of the hour is when most users login to a call. What if you have a two-person conference call or a 10-person conference call? Which ones are most likely? What happens when everyone exits the software (video conferencing) at the same time?
- What functions do they consider most important? Beyond logging in, do you have a priority scheme based on different user types? What is most important to each user role and are these tasks optimized for performance?
Based on the answers to these questions, pick out the most difficult from a simple manual execution standpoint. You may see latency in certain areas just by doing some of these scenarios manually. Evaluate them simply by slow, medium and fast, then cross reference with frequency of use and importance of that function. From there, develop performance scripts by priority using the schema above. That information and the results can then be used for benchmarks.
As you can see, when it comes to performance testing considerations, the first step is figuring out what is important to you in terms of performance? Does it really matter if a rarely used function has a response time of under 1 second? Don’t jump to the tools and worry about getting results out of the gate until you’re clear on how you’ll analyze the results , how you’ll track and record the data, and what you’ll do with them. By using a methodical approach, you’ll be much better off in the long run with results you can compare over time and use as a basis for improvement.
Step #2 – Leveraging Google Analytics
As mentioned in Step 1, understanding what users are doing and when, is paramount before embarking on a performance testing effort. One of the most common tools for doing this is Google Analytics (GA). GA is free of charge, providing many useful metrics like GA Pageviews Report and GA Behavior Flow Report (for additional details, see our webinar “Performance Testing Considerations Using JMeter and Google Analytics”). We believe there is plenty of useful information just within the free reports, however, Google also has available enhanced Enterprise/Business versions that can be purchased as an upgrade.
We listed out some 3 question and answer scenarios that we often come across in our consulting engagements.
1. Q: Do you know of any resources that can help testers learn Google Analytics?
A: There are many resources on the Internet that provide additional information on GA. The Google Analytics Solutions pages provides a in depth information on the available reports. There are also e-learning sites (e.g., udemy.com, lynda.com) that provide tutorials on GA, from beginner to advanced. Remember that GA provides a lot of great information for marketing. However, for our load/performance test focus, we use GA to ensure that we are focused on gathering information on application areas that are heavily utilized, either now or in the future. We use that information to develop our Performance Prioritization Matrix to set priorities.
2. Q: How do you read the stats for Google Analytics? For example, under the starting pages, you have 65.1K sessions and 63.1K drop-offs. What exactly do you mean by drop-offs? Are they canceling (backing out) or just moving on to the next section of the application?
A: The drop-offs are a “backing out” of the session. For example, of the 65,100 sessions, only 2,000 users continued to the “1st interaction”. Also, it is possible to define paths through the site, so that if a defined path is not pursued by the user, that would be flagged as well. Specific definitions for each of available statistics are provided on the Google Analytics Solutions pages.
3. Q: A big challenge is how to define concurrent users. Normally, the client believes that he needs to inject the total population that he has or that he is going to have. What is the best way to calculate that?
A: This is a complex, very interesting question. There are a few factors that need to be calculated. Basically, you need to figure out how many people are using the application across a given short time period (e.g., busiest one hour). GA can do this easily. Then you need to figure out how long they are on the site. This is also reasonably easy to do using GA (GA Pageviews). You can then apply a simple formula:
Avg Concurrent Users = (Total Peak Hr Visitors) * (Avg Time in sec on Page) / 3600
However, this is the average number of concurrent users. What we really want is the peak number of users “visiting” the site, or executing a particular function, at the same time. Without getting into all the statistics, a rule of thumb could be Peak Utilization factor of 1.5 or 2 can be applied. So, if there were 300 concurrent users, a value of 600 concurrent users could be utilized as a starting place for peak simultaneous load value evaluations. Again, you have to consider what they are doing, you may have 200 concurrent users printing a report that is database intensive, in combination with other groups of users using other functions of the software. You’ll need to understand each of these performance characteristics independently as well as in conjunction with other simultaneous user groups.
Step #3 – JMeter Tips
Once you’ve figured out what users are doing, what data you’ll collect, how you’ll analyze that data and what you’ll do with the results, then it’s time to dig in and figure out how to get the performance data. This, of course, involves using a tool combined with designing and executing performance tests. One of the most common open source tools in JMeter.
Q: Is there a better way to describe master/slave tasks? I find that language problematic. Main/secondary?
A: The terms “master” and “slave” have been a standard reference in engineering parlance in the industry for decades. Most of the industry has been moving toward updating these terms. Common replacement terms are leader/follower, parent/child, and master/replica. For our needs, we will be using master/replica below.
In the case of JMeter, the terminology in the webinar reflects that the master provides control over the replicas by informing the replica machines what scripts to run, as well as serving as a central source for the collection of resulting data.
Q: Can replicas be VMs or do they need to be physical machines?
A: Replicas do not need to be physical machines. They can be virtual machines. In this regard, JMeter can really take advantage of the ever-increasing availability, configuration, and location flexibility offered by cloud-based computing services like Amazon’s AWS and Microsoft’s Azure.
Q: How does one go about simulating replicas from different geo locations?
A: Configuring a master/replica configuration over geographic locations can be tricky. The key is that the master and replica machines must be configured utilizing public IP addresses. Also, the master will need to be configured to instruct the replicas on what tests to run, as well as what information to return to the master for post processing and analysis.
Q: With older versions of JMeter, when we schedule a load test with multiple load gen, we had to manually calculate ramp-up time to keep it uniform. Is that still the case? Also, reporting was pretty primitive before, what changes have been made?
A: With regard to ramp-up time, the Stepping Thread Group plugin (referenced in the webinar) is one that we often use to generate load of increasing intensity. That plugin, as well as others, are available at jmeter-plugins.org. There are also a host of plugins that provide a broad range of reports. JMeter’s extensive plugin catalog is one of the reasons that XBOSoft has focused on JMeter as our performance tool of choice.
Q: What would be the resources needed of every PC that will be a JMeter injector and the number of users in order to calculate the number of computers that we need to execute and the injectors work without blockers?
A: Metrics from the server under test can be monitored through various means. The “PerfMon” plugin on the JMeter master is a popular method. Additionally, a server agent needs to be installed within the testing environment. The agent can be installed as well on the application test server or the JMeter replicas. All statistics are be fed back to the JMeter master.
Some sites quote 300 as the maximum number of threads that can be run by any given JMeter instance. However, the amount of load that will be placed on the JMeter replicas will depend on the types of performance scripts being run, script complexity, RAM and CPU capacity, of the replica machines, etc. You can use 300 as a rule of thumb to start, but our recommendation is to monitor the actual CPU and memory loads, as well as network throughout replica machines, during the times that the scripts are being executed, ensuring that there is headroom throughout the performance test execution.
As mentioned above, one of the reasons we chose JMeter as our tool of choice when conducting performance testing is the vast plugin library available that enable you to configure JMeter to do just about anything you want, including API testing, as well as some fairly sophisticated reporting and analysis.