Introduction
Primarily, in the software industry, there are two kinds of testing. They are functional testing and non-functional testing. Non-functional testing is also known as Performance testing and within performance testing, it is further classified into baseline and benchmark testing in performance testing. In this article, we are going to explore in detail non-functional testing esp. benchmark testing.
Any software application enters into its various testing phases such as unit testing, integration testing, system testing, and acceptance testing. Where we cover most of the aspects of the white box testing i.e. unit testing and black-box testing or functional testing. What about the performance of a software application? Can we say that an application that is functionally sound after testing is fit for the business purpose? Definitely not. There is another important aspect of testing which is known as non-functional testing or performance testing. It is a type of testing that evaluates the responsiveness and the stability of a software application in terms of its load capacity, behavior under stress conditions, reliability, scalability, etc. When any software application is developed, we always capture the requirements into Functional requirement document (FRD) such as the volume of request per unit time to the application from end-user, service level agreement, permitted delay in response from the application server, load condition, etc. If a software application is functionally sound and its performance criteria are well satisfied after carrying out performance testing, then we can conclude that the software application is ready to use i.e. eligible for production.
What are baseline and benchmark testing?
When any software application undergoes performance testing, there we record the performance testing metrics in terms of various application aspects. After a certain period of time, when the same application undergoes the change such as operating system upgrade, hardware upgrade, patch, etc. then the performance testing team executes the test to determine the new performance testing metrics and compare the collected metrics against the baseline metrics which was collected and documented before. This kind of testing is known as baseline testing.
Benchmark testing, on the other hand, is the company-wide agreed standard for the performance testing metrics which every software application that belongs to that organization should pass. Preferably, the baseline performance metrics values should not exceed benchmarked performance standards.
A key rule for a benchmark is it should be repeatable. For example, while doing different load testing, if the response time varies every time then system performance testing should be benchmarked. The response time on all conditions load should be constant.
Also, the benchmark should be quantifiable. We can’t be quantified UI experience in numbers, however, we can quantify the time user spent on good webpage UI.
What is performance Metrics?
When we conduct the performance testing for any software application, then we usually record the following performance aspects.
Web Server Performance Metrics:
Most of the software applications which are used by various organizations are web-based. Any web-based application uses the webserver such as Apache, IIS, Nginx, etc. The following are the performance metrics that are determined under the performance test.
- Busy and Idle Threads: Web application serves the request by assigning them to the server threads which have the capability of executing the request in parallel i.e. multithreading environment. To evaluate this performance meter, we load the server with the required number of requests and then test the performance after calculating the number of worker thread idle threads, etc. against the delay in the server response.
- Throughput: It is the calculation of the number of requests served per unit time i.e. transactions per second or minute by the webserver. It defines the scalability of the web application.
- Bandwidth requirements: Under this performance parameter, we evaluate the bottleneck in the network, weight of the web pages while loading, content to CDNs, etc. Accordingly, we decide the network bandwidth under which the application is going to operate in an efficient way.
App Server Performance Metrics:
A web application is built on the following platforms: Java, .NET, PHP, etc., each of these platforms has the parser or the environment set up on the application server. The following are the key performance metrics that determine the deployment or configuration problems on these application servers.
- Load Distribution: It is the measure of the volume of the transactions which are handled by each JVM (in JAVA), CLR (in .NET) and PHP engine. Next, we check if these transactions or the requests to the app server properly load-balanced or more application servers are required to handle the load (clustered).
- CPU Hotspots: Here we evaluate the application performance based on the available system CPU (Central processing unit) on the application server. We check for the CPU utilization on the application server and If CPU utilization is more all the time and the delays are caused in the responses then we might have to consider to increase the CPU power. Sometimes more CPU consumptions are because of the bad programming such as unnecessary loops in the code, dead code or evaluated values that are not in use, unnecessary waits in the program, etc. Such loopholes in the code should come up in code reviews.
- Worker Threads: It is the measure of the volume of the worker threads which are correctly configured. How often these threads are busy due to web server unavailability under load or stress condition and identifying the webserver modules that block these threads.
- Memory Issues: This performance parameter deals with the identification of the bad memory patterns, identification of the memory leak, impact of the garbage collection on the CPU and Transaction Throughput as these threads are continuously operating in the background while worker threads are operating, etc.
Host Health Performance Metrics:
Both Web and Application Servers run on the physical or the virtualized hosts. Therefore, it is very important to do a performance sanity check on all hosts that are involved in the application infrastructure. It includes the following key metrics.
- CPU, Memory, Disk, Input/Output: It is the measure of the CPU, memory, disk and I/O usage when the application operates in a production-like environment. It should not happen that under the normal load condition any of these resources get exhausted. We also check for the volume of the logs that the application is writing to the disk to avoid flooding the disk.
- Key Processes: It is the measure of the types of processes that run on the box. Which processes are consuming the number of resources or creating the deadlock, etc.? At this performance testing step, we make the decision either to increase the number of resources or move the processes to another box.
Software Application Performance Metrics:
We are also required to test the performance of the application code under execution. The following are the key metrics to test the application code performance.
- Time Spent in Logical Tier or Layer: The enterprise applications are multi-tier e.g. MVC (Model View and Controller) application. Here, we check which tier is taking more time for execution than others with the increasing load on the application. This helps in understanding the need to scale that tier of the application which is the sole creator of the hurdle in the application operation.
- Number of Calls into a Logical Tier or Layer: It is the measure of the calls to the internal Web Services and other critical APIs such as Hibernate, Spring, Struts, etc.
Conclusion
In the article, we have covered the key performance metrics along with the justification of what they are important to test for a software application as a baseline and benchmark testing. Also, there could be other performance metrics that depends on the nature of an application, technology platform, end-user, etc. In the upcoming article, we will discuss benchmark testing tools or softwares.
⇓ Subscribe Us ⇓
If you are not regular reader of this website then highly recommends you to Sign up for our free email newsletter!! Sign up just providing your email address below:
Happy Testing!!!