1. Q: Do you know of any resources that can help testers learn Google Analytics?
A: There are many resources on the Internet that provide additional information on GA. The Google Analytics Solutions pages provides a in depth information on the available reports. There are also e-learning sites (e.g., udemy.com, lynda.com) that provide tutorials on GA, from beginner to advanced. Remember that GA provides a lot of great information for marketing. However, for our load/performance test focus, we use GA to ensure that we are focused on gathering information on application areas that are heavily utilized, either now or in the future. We use that information to develop our Performance Prioritization Matrix to set priorities.
2. Q: How do you read the stats for Google Analytics? For example, under the starting pages, you have 65.1K sessions and 63.1K drop-offs. What exactly do you mean by drop-offs? Are they canceling (backing out) or just moving on to the next section of the application?
A: The drop-offs are a “backing out” of the session. For example, of the 65,100 sessions, only 2,000 users continued to the “1st interaction”. Also, it is possible to define paths through the site, so that if a defined path is not pursued by the user, that would be flagged as well. Specific definitions for each of available statistics are provided on the Google Analytics Solutions pages.
3. Q: A big challenge is how to define concurrent users. Normally, the client believes that he needs to inject the total population that he has or that he is going to have. What is the best way to calculate that?
A: This is a complex, very interesting question. There are a few factors that need to be calculated. Basically, you need to figure out how many people are using the application across a given short time period (e.g., busiest one hour). GA can do this easily. Then you need to figure out how long they are on the site. This is also reasonably easy to do using GA (GA Pageviews). You can then apply a simple formula:
Avg Concurrent Users = (Total Peak Hr Visitors) * (Avg Time in sec on Page) / 3600
However, this is the average number of concurrent users. What we really want is the peak number of users “visiting” the site, or executing a particular function, at the same time. Without getting into all the statistics, a rule of thumb could be Peak Utilization factor of 1.5 or 2 can be applied. So, if there were 300 concurrent users, a value of 600 concurrent users could be utilized as a starting place for peak simultaneous load value evaluations. Again, you have to consider what they are doing, you may have 200 concurrent users printing a report that is database intensive, in combination with other groups of users using other functions of the software. You’ll need to understand each of these performance characteristics independently as well as in conjunction with other simultaneous user groups.
Q: Is there a better way to describe master/slave tasks? I find that language problematic. Main/secondary? A: The terms “master” and “slave” have been a standard reference in engineering parlance in the industry for decades. Most of the industry has been moving toward updating these terms. Common replacement terms are leader/follower, parent/child, and master/replica. For our needs, we will be using master/replica below. In the case of JMeter, the terminology in the webinar reflects that the master provides control over the replicas by informing the replica machines what scripts to run, as well as serving as a central source for the collection of resulting data.
Q: Can replicas be VMs or do they need to be physical machines? A: Replicas do not need to be physical machines. They can be virtual machines. In this regard, JMeter can really take advantage of the ever-increasing availability, configuration, and location flexibility offered by cloud-based computing services like Amazon’s AWS and Microsoft’s Azure.
Q: How does one go about simulating replicas from different geo locations? A: Configuring a master/replica configuration over geographic locations can be tricky. The key is that the master and replica machines must be configured utilizing public IP addresses. Also, the master will need to be configured to instruct the replicas on what tests to run, as well as what information to return to the master for post processing and analysis.
Q: With older versions of JMeter, when we schedule a load test with multiple load gen, we had to manually calculate ramp-up time to keep it uniform. Is that still the case? Also, reporting was pretty primitive before, what changes have been made? A: With regard to ramp-up time, the Stepping Thread Group plugin (referenced in the webinar) is one that we often use to generate load of increasing intensity. That plugin, as well as others, are available at jmeter-plugins.org. There are also a host of plugins that provide a broad range of reports. JMeter’s extensive plugin catalog is one of the reasons that XBOSoft has focused on JMeter as our performance tool of choice.
Q: What would be the resources needed of every PC that will be a JMeter injector and the number of users in order to calculate the number of computers that we need to execute and the injectors work without blockers? A: Metrics from the server under test can be monitored through various means. The “PerfMon” plugin on the JMeter master is a popular method. Additionally, a server agent needs to be installed within the testing environment. The agent can be installed as well on the application test server or the JMeter replicas. All statistics are be fed back to the JMeter master.
Some sites quote 300 as the maximum number of threads that can be run by any given JMeter instance. However, the amount of load that will be placed on the JMeter replicas will depend on the types of performance scripts being run, script complexity, RAM and CPU capacity, of the replica machines, etc. You can use 300 as a rule of thumb to start, but our recommendation is to monitor the actual CPU and memory loads, as well as network throughout replica machines, during the times that the scripts are being executed, ensuring that there is headroom throughout the performance test execution. As mentioned above, one of the reasons we chose JMeter as our tool of choice when conducting performance testing is the vast plugin library available that enable you to configure JMeter to do just about anything you want, including API testing, as well as some fairly sophisticated reporting and analysis.