Load Testing Framework

Hi, while preparing a paper on Load Testing for the internal journal ;I found a very useful article while goggling  related to Load Testing Framework; I had summarized that in a document; for more information you can visit http://msdn.microsoft.com/.

Load Testing Framework

Objective:

Objective of this article is to provide a framework which can be used as a basic approach for performing load testing.

Understanding the Key Concept:

Load testing is defined as the process of determining the behaviour of system (application) and servers (website) by feeding it with the largest tasks it can operate with.Load testing is sometimes called volume testing.

Ø      Examples of volume testing:

·         testing a word processor by editing a very large document

·         testing a printer by sending it a very large job

·         testing a mail server with thousands of users mailboxes

Goals of Load Testing?

The basic purpose of load and performance testing an application and/or website is to accomplish the following:

• To measure the effects on user experience as total user load is increased
• To measure the average page response time and determine if the performance is acceptable at both average and maximum user loads
• To quantify the upper limit of the hardware being accessed by the application to determine whether the current system is capable of handling the anticipated future loads.

What functions or business processes should be tested?

The following table describes the criteria for determining the business functions or processes to be included in a test.

Basis for inclusion in Load Test	Comment
High frequency transactions	The most frequently used transactions have the potential to impact the performance of all of the other transactions if they are not efficient.
Mission Critical transactions	The more important transactions that facilitate the core objectives of the system should be included, as failure under load of these transactions has, by definition, the greatest impact.
Read Transactions	At least one READ ONLY transaction should be included, so that performance of such transactions can be differentiated from other more complex transactions.
Update Transactions	At least one update transaction should be included so that performance of such transactions can be differentiated from other transactions.

     Load Test Framework:

     Identify Load test acceptance Criteria

First we need to identify performance acceptance criteria; it is most valuable when initiated early in the application’s development life cycle.
Test objectives frequently include the following:

• Response time.  For example, the transaction page must be displayed in less than 3 seconds.
• Throughput.  For example, the system must support 100 transactions per second.
• Resource utilization.  A frequently overlooked aspect is the amount of resources your application is consuming, in terms of processor, memory, disk input output (I/O), and network I/O.
• Maximum user load.  This test objective determines how many users can run on a specific hardware configuration.
• Business related metrics.  This objective is mapped to business volume at normal and peak values; for example, the number of orders or Help desk calls handled at a given time.

     Identify Key Scenarios

1. Identify all the scenarios for a Web application. For example, The basic scenario for an e-commerce website will be:

• Browse catalogue
• Search for a product
• Place an order

2. Identify the activities involved in each of the scenarios. For example, a “Place an Order” scenario will include the following activities:

• Log on to the application.
• Browse the product catalog.
• Search for a specific product.
• Add items to the shopping cart.
• Validate credit card details and place an order.

3. Identify the scenarios that are most commonly executed or most resource-intensive; these will be the key scenarios used for load testing. For example, in an e-commerce application, browsing a catalog may be the most commonly executed scenario, whereas placing an order may be the most resource-intensive scenario because it accesses the database.

• The most commonly executed scenarios for an existing Web application can be determined by examining the log files.
• The most commonly executed scenarios for a new Web application can be obtained from market research, historical data, market trends, and so on.
• Resource-intensive scenarios can be identified by using design documents or the actual code implementation. The primary resources are:
• Processor
• Memory
• Disk I/O
• Network I/O

      Create a Workload Model

After identifying the scenarios we will create the work model, creating a work model means understanding the user’s behaviour, how a user can perform all these scenarios’.

• A user scenario is defined as a navigational path, including intermediate steps or activities, taken by the user to complete a task. This can also be thought of as a user session.
• A user will typically pause between pages during a session. This is known as user delay or think time.
• A session will have an average duration when viewed across multiple users. It is important to account for this when defining the load levels that will translate into concurrent usage, overlapping users, or user sessions per unit of time.
• Not all scenarios can be performed by a new user, a returning user, or either; know who we expect your primary users to be and test accordingly.

     Identify Target Load Levels

Identify the load levels to be applied to the workload distribution(s) identified during the previous step. The purpose of identifying target load levels is to ensure that your tests can be used to predict or compare a variety of production load conditions. The following are common inputs used for determining target load levels:

• Business volume (both current and projected) as it relates to your performance objectives
• Key scenarios
• Distribution of work
• Session characteristics (navigational path, duration, percentage of new users)

By combining the items above, we can determine the remaining details necessary to implement the workload model under a particular target load.

     Identify Metrics

There are many measurements that you can use when load testing. The following metrics are key performance indicators for your web application or web site.

Ø      Average Response Times

Ø      Peak Response Times

Ø      Error Rates

Ø      Throughput

Ø      Concurrent Users

Average Response Times

When you measure every request and every response to those requests, you will have data for the round trip of what is sent from a browser and how long it takes the target web application to deliver what was needed.
Peak Response Time
Similar to the previous metric, Peak Response Time is measuring the round trip of a request/response cycle. However the peak will tell us what is the LONGEST cycle at this point in the test.
The Peak Response Time shows us that at least one of our resources are potentially problematic. It can reflect an anomaly in the application where a specific request was mishandled by the target system. Usually though, there will be an "expensive" database query involved in fulfilling a certain request such as a page that makes it take much longer, and this metric is great to expose those issues.
Error Rate
It is to be expected that some errors may occur when processing requests, especially under load. Most of the time you will see errors begin to be reported when the load has reached a point that exceeds the web application's ability to deliver what is necessary.
Throughput
Throughput is the measurement of bandwidth consumed during the test. It shows how much data is flowing back and forth from your servers.
Throughput is measured in units of Kilobytes Per Second.
Concurrent Users
Concurrent users are the most common way to express the load being applied during a test. This metric is measuring how many virtual users are active at any particular point in time.

     Design Specific Tests

Using your scenarios, key metrics, and workload analysis, we can now design specific tests to be conducted. Each test will generally have a different purpose, collect different data, include different scenarios, and have different target load levels. The key is to design tests that will help the team collect the information it needs in order to understand, evaluate, or tune the application.

     Run Tests

Consider the following steps when preparing to simulate load:

1. Configure the test environment in such a way that it mirrors your production environment as closely as possible, noting and accounting for all differences between the two.
2. Ensure that performance counters relevant for identified metrics and resource utilization are being measured and are not interfering with the accuracy of the simulation.

     Analyze the Results

We can analyze the test results to find performance bottlenecks between each test run or after all testing has been completed.
The following are the steps for analyzing the data:

1. Analyze the captured data and compare the results against the metric’s accepted level to determine whether the performance of the application being tested shows a trend toward or away from the performance objectives.
2. Analyze the measured metrics to diagnose potential bottlenecks. Based on the analysis, if required, capture additional metrics in subsequent test cycles. For example, suppose that during the first iteration of load tests, the process shows a marked increase in memory consumption, indicating a possible memory leak. In the subsequent iterations, additional memory counters related to generations can be captured to study the memory allocation pattern for the application.