Writing Performance Tests in Grinder using a Framework

This project is now available on GitHub.

Before reading this tutorial (if you haven’t already), please took at look at my last two tutorials (Performance Testing using The Grinder and Anatomy of a Grinder test-script). In this tutorial I’ll talk about easily writing Grinder test-scripts using a framework I designed. As a disclaimer, I’d like to point out that I’m not a Python programmer and therefore certain things may not be very python-esque. If that’s the case, I apologize. My personal opinion is that this framework is especially useful (of course, since I wrote it ;)) for web applications where you have a already have a lot of test data. In that case, you can simply record all your discrete tasks once and then construct different scenarios with them. But if you feel differently and have some constructive criticism, I do look forward to hearing from you! Also, if you’d like to try out the framework I’ve got a tarball and a zip file available for download on the very last page.

The motivation for a framework

Typically during performance-testing, you want to test different kinds of scenarios. These scenarios are made up of discrete tasks (also known as transactions), for example, consider a scenario where a user logs in, searches for a person, and then logs out. In this scenario, you have three separate tasks: logging in, searching for a person, and logging out. Using Grinder’s TCPProxy tool, you can easily record this scenario and you can also parameterize it. What happens when you want to record another scenario? Say, one that involves logging in, searching for a person, adding a new person, and logging out? Sure, you can go ahead and record that, and even parameterize it. But consider the fact that the new scenario is a super-set of the old scenario. It has three tasks in common with the first scenario. What if there was a way to create new scenarios from scratch, not by recording, but by using previously recorded tasks? This way you would only have to record information once, and after that, you can reuse it. To do this, we need to first understand how to identify tasks within a recorded Grinder test-script. Using this as our foundation, we can figure out how to pull out the tasks into discrete units, which we can later reuse.

The Relationship between Requests and Tasks

In the previous tutorial, I went over the anatomy of a recorded Grinder test-script. There, we saw that the script defines a bunch of requests, and then has methods that correspond to each recorded page. In each method, the script executes a bunch of requests. Initially you might think that we would simply pull out these individual methods. However, that is not the case. A single task can involve more than one recorded page. For example, the act of logging into the app involves accessing the login page, then logging in, and then hitting whatever page the login process drops you into. This involves (at the very least) three pages. In fact, the relationship between requests, pages, and tasks looks something like this:

Here, you can see that a set of unique requests belong to a recorded page, and then a set of recorded pages belong to a unique task. Finally, a set of tasks belong to a scenario. What we want to do is pull out the individual tasks so that we can reuse them to create different scenarios. To do this, we will still be using Grinder’s TCPProxy tool (at least to record the scenario), but our resulting script will not look like the typical Grinder test-script. It will, instead, conform to the testing framework.

A high-level view of the framework

To understand the framework, first let’s figure out what it is that we want to do exactly. To create a scenario, our framework should let us do the following things:

How about looking at what we want to do from a programming perspective?

Of course, this is still a rather simple view. We actually want to do a bit more than that. We also want to parameterize our tasks. So, the list of things we want to do actually looks like this:

Now that we know what we want to do, let’s take a look at the actual code behind the framework.

The Task class

All tasks that you record and create will extend the Task class. The Task class by itself is pretty simple, and not very useful. It looks like this:

Task.py:

Although not useful by itself, the Task class does define some important methods and properties:

Class Variables

In Python, properties defined at the top of the class are called Class Variables. They are different from Instance Variables in the sense that they are shared between all instances of a class. The numberOfTasks property holds the number of Task instances that have been instantiated. The reason we need to maintain this property will become clearer later.

Constructor init is the constructor for the Task class. Here we initialize a few instance properties. The first property is the urlDict property, which maintains a dictionary of unique URLs. If you’ll remember, when I talked about the structure of a test-script, I mentioned that the recorder keeps track of unique URLs that it encountered during the recording process. It then used the URLs and headers to create request objects. In a typical recorded-script, the URLs are simply defined as global variables within the script. Here, they are maintained as key-value pairs with the urlDict dictionary.

The next property is the parameterizingMethods property. This property is pretty important since it is what allows us to parameterize requests. I won’t go into too much detail; for now just realize that it keeps tracks of methods we use to parameterize our requests.

At the end of our constructor, we increment the value of numberOfTasks by one, because as I mentioned before, this property keeps track of the number of instances of this particular class, that have been instantiated.

Accessor methods

The Task class has three self-explanatory accessor-methods which are getUrlDict, getUrl, and getParameterizingMethodFor. The first method returns the entire urlDict dictionary, while the second returns a particular URL (identified by a supplied key) from the urlDict dictionary. The third method returns the parameterizing method for a particular (supplied) key.

Mutator methods

The Task class also has three self-explanatory mutator methods which are setUrlDict, setUrl, and setParameterizingMethodFor. The first method sets the urlDict property to the one supplied, while the second one sets the value of a particular URL within the urlDict dictionary (identified by a supplied key) to the supplied value. The setParameterizingMethodFor method sets the parameterizing method for a particular (supplied) key.

Utility methods

The Task class has two utility methods. The first one, callParameterizingMethodFor, calls the parameterizing method identified by the supplied key. The second method, instrumentMethod is quite similar to the instrumentMethod method in a typical recorded-script. The only difference is that there is no default argument, because we don’t need one in this case (instrumentMethod is a class method now, as opposed to a global method).

Now that we’ve gone over the base class, let’s take a look at a class that has been derived from the Task class so that all of this makes a bit more sense. While going through it, compare it to the structure of a typical recorded test-script. You will notice a few similarities. *Note: *In the following code you’ll notice that the framework is called torqueo. I called it that because the word means "to twist, curl, rack, torture, torment, distort, or test" in Latin. I think that accurately describes what we’re trying to do in performance testing. Yes, I’m a dork.

LoginTask LoginTask is a relatively simple task. All it does is log into the application. It looks like this:

LoginTask.py

At first glance, this script appears to be rather similar to a recorded grinder-test-script, but there are a few important differences:

The first thing to note is that the headers are not global variables. Instead, they are class variables now. If you’ll recall, the TestRunner class in the recorded test-script had methods that corresponded to each recorded page. The situation is similar here. The LoginTask class also has methods that correspond to each recorded page (you might be wondering why they are called 'appLoginN' instead of the non-descriptive 'pageN', but I’ll get to that later). Finally, instead of the call method, LoginTask has a run() method (in retrospect, I guess I could have made the class callable, but it didn’t seem necessary. I’m also note that familiar with Python and I wanted to avoid any unpleasant surprises).

Now let’s go over the important aspects of this class:

Constructor

The constructor for LoginTask is pretty simple. In the very first line, we call the parent class’s constructor, and then we give this class a description. In the second line, the urlDict property is set to an empty dictionary. You might wonder why we aren’t actually setting any values for the URLs. The reason is that we want the parent Scenario to define the URLs for its child tasks. Therefore, the tasks themselves will not have any default URLs. Instead, they will inherit the URL values from the Scenario that they belong to. The last line is pretty important. You’ll recall that in the constructor of the Task class, the class variable numberOfTasks was incremented by one. Here, we take that value and assign it to the taskId attribute of LoginTask. What we’re essentially doing is giving each instantiated task a unique id. You’ll see why that’s important in a little bit.

initializeTask

This is probably the most important method. Here we set up data for our task and strangely enough, initialize our instance. The first thing we do is set up our requests. You’ll notice that the manner in which we do this is pretty similar to the way that the recorded test-script does it. However, in our case, for the URL values we use the urlDict dictionary, and the headers are class instance variables. Another important thing to notice is the first argument to the Test class’s constructor. You’ll notice that it is an expression that includes self.taskId. The reason we use the id of the task is because it is unique. Tests in Grinder need to have unique numbers. Remember that we want our tasks to be reusable. So it is conceivable to have a situation where we are using more than one instance of LoginTask. If we didn’t give each test a unique id and simply used the format [pageNumber][requestNumber] then there would be no way for Grinder to distinguish between tests and our results wouldn’t make very much sense. Therefore, our test numbers will be of the format [taskId][pageNumber][requestNumber]

After initializing our requests, we set up the parameters dictionary. This dictionary is important in two ways. First, it maintains a list of name-value pairs that are used in POSTs and GETs, and second, it helps us parameterize those values (by letting us directly modify it). In the initializeTask method, the parameters dictionary is initially set up with values collected by the recorder. This way, we have sensible defaults. The parameters dictionary is a 3-level dictionary. The very first level has keys that correspond to method names within the class. The second level consists of keys that match up with request numbers. The final level contain the parameter names themselves (used in GETs or POSTs). So, to access a parameter value you essentially do this: parameters[methodName][requestNumber][parameterName]. To understand the rationale behind this design, think back to the diagram that describes the relationship between requests and tasks. A set of requests belong to a page, so it makes sense to have the top level of the hash to be method names (since they correspond to each recorded page). Then, the bottommost level will contain parameter names that belong to a particular request.

The last set of statements are similar to the calls to instrumentMethod in a typical recorded-script. Notice, however, that once again we use the taskId property to ensure that we have unique test numbers.

Page methods

Similar to a typical recorded test-script, the page methods of the LoginTask class correspond to each unique page that was recorded during the task. The code in these methods is very similar to the one in a recorded test-script. However, in the original script, parameter values within a querystring (for GETs) were pulled out into "tokens" and were turned into instance variables of the TestRunner class. For example, the querystring parameter msg would become self.token_msg. In our framework, however, the querystring parameters and their default values are stored in the parameters dictionary and accessed the same way that POST parameters and values are.

run() method

The run() method is similar to the call() method in a recorded test-script. The only difference is that call the callParameterizingMethodFor method before we actually call the page method. You should now be able to see how we can easily parameterize the values used by requests in a page method. The parameterizing method will modify the values in the parameters hash, and these values will be used by the requests in the corresponding page method.

Hopefully you now have a clearer picture of the framework. You should now be able to see how we can design tasks to be discrete units that we can reuse, customize, and parameterize to our wishes. Even better: you don’t have to write the code for these tasks from scratch. I’ve written a translator script in Perl that converts the XML data recorded by TCPProxy and converts it into a class like the one above (I’ll talk about that later). Now that you’ve seen how a task is designed, let’s go up one level and take a look at the Scenario class.

The Scenario class

The Scenario class (as the name suggests) describes a particular scenario. It contains a list of tasks that it will execute sequentially. It looks like this:

Scenario.py

Constructor

The constructor for the Scenario class accepts two arguments: a description and a dictionary for URLs. As mentioned earlier, each task has a URL (or URLs) associated with it. By setting a URL for a scenario, you cause all tasks added to that particular task to have the same URL. Simce the tasks by themselves do not have any previously defined URLs, we need to make sure that the supplied dictionary is not empty. The constructor also initializes a list called tasks which of course, maintains a list of tasks.

addTask method

The addTask() method lets you add a Task to a Scenario. First we check to see if the object passed in implements the setUrlDict() method. If it doesn’t we raise an exception. If it does implement the method, we check to see whether the current instance of the Scenario class has a non-empty urlDict dictionary. If it does, we go ahead and set the task’s urlDict property to the same value. As a side note, Python doesn’t have explicit interfaces. The language has implied interfaces. Hence, we need to inspect the object at runtime to see whether it implements a necessary method.

The next if-statement checks to see whether the supplied object implements the initializeTask() method. If it does, we go ahead and run that method.

Finally, we check to see if the supplied object implements the run() method, since it makes no sense to add a task that we cannot run. If it does, we add it to our list of tasks.

Accessor methods

The class has two accessor methods: getUrl() and getUrlDict() which perform the same functions as the Task base classes’s getUrl() and getUrlDict() methods.

Mutator methods

You might be wondering why the mutator methods don’t actually let you change anything. The reason is because all the child tasks of a Scenario must have the same URL. As you can see, we initialize the URL of a task only when it is added. If you were allowed to change the URL of a Scenario later, then the tasks wouldn’t have the same URL as the scenario. Of course, you could then say that in the setUrlDict() or setUrl() methods we just need to update all the tasks with the new URL. But that would be a side-effect, and side-effects are not good! Finally, there simply is no reason to change the URL of a Scenario during runtime. Since the tests themselves are tied to a task, and therefore to a scenario, altering the URL of a scenario while the test is running would make your statistics meaningless.

run() method

The run() method iterates through all the tasks belonging to this scenario.

Now that we’ve gone over the building blocks of the framework, let’s use them to create a scenario that we want to test.

A Login and Logout Scenario

Our first scenario is going to be a very simple one. We’re just going to log into the application and then log out. We’re going to use default values and aren’t going to parameterize anything:

login-logout.py

Pretty simple, right? First we create an instance of a Scenario. We give it a description and also a dictionary containing the URL we’re going to be using. Then, we create an instance of a login task, a logout task, and add it to our scenario. Finally, we run our scenario in the call method. Now what if we want to parameterize the logins? That’s not difficult either:

login-logout-parameterized.py:

Here we do two additional things. After creating an instance of LoginTask, we create an array of name-value pairs and assign it to loginCredentials. We then create a new property for this particular instance of LoginTask called loginCredentials, and set the value of that property to the array we just defined. The reason we do this is because we’re going to be refering to this array in our parameterizing method. Then, we actually define the parameterizing method. In the method, we choose a name-value pair out of the array using the current thread number as an index. We then use the username and password in the name-value pair as our login credentials. What we’re essentially doing is modifying the username and password parameters for request number 201 in the appLogin2 method of LoginTask. The neat thing is that you can put whatever logic you want inside the parameterizing method. You could even read from a file if you wanted to.

Finally, here is one more example. It’s a bit more complicated, but see if you can figure out what it’s trying to do. I’m providing a link to it (/api/media/5e9b98e5-60e1-453c-bce0-8fa1f5bcadf4/content]) since it’s a little long. The code is commented so that should help :): I hope this demonstrates the power and flexibility of the testing framework. Now you’re ready to write your own tests using this framework! But before that, you’ll need to know how to translate the data from TCPProxy.

Using xmlToJython.pl /api/media/5e9b98e5-60e1-453c-bce0-8fa1f5bcadf4/content] is a handy script that I wrote, that translates XML header data from TCPProxy into a Jython class. Before you use it, you need to have TCPProxy return XML data instead of Jython. You will need to create a new shellscript:

Then, you’ll need to use xmlToJython.pl to convert the XML file into Jython. The syntax for xmlToJython.pl is this:

The output file is optional. If one is not provided, the script just outputs to STDOUT. The script requires that the XML::Simple module has been installed. The easiest way to do that is through cpan (a commandline tool to install Perl modules). I’m going to see if I can compile the script into a binary so that you don’t have to jump through hoops to use it.

A few more tips…​ xmlToJython.pl has a feature that lets you make your generated script a bit nicer to read. In the code samples above, you’ll notice that the methods had nice names like "appLogin1" instead of "page1", and that the description strings for the tests were actually informative instead of just "GET /somefile.html". Here’s how you can do that: When you start up TCPProxy, you’ll notice that you can add comments. If you make sure that your very first comment is of the form:

Then, your recorded class will be called NameOfTaskClass, your page methods will start with pageMethodName, and your test descriptions will have "Description of Task" prepended to them. Typically you should only need that very first comment since you’re only recording one discrete task.

Package Structure

For the framework, I created a package structure (you can see it in the way I import the task files). Within my performance-testing directory where I have Grinder installed, I created a scripts directory, inside which I place the package:

Of course, you can create it in any manner you see fit; this is just the way I did it.

Well, that’s the end of this tutorial! Hope it was informative and helpful! Please comment and let me know where I can improve. This was my first time doing a lot things; it was my first time writing a framework, my first time doing performance testing, and my first time doing anything serious (and by serious I mean more serious than "Hello World!") in Python/Jython.

Download

If you want to check out the framework, here are the archives. They contain the Task and Scenario classes, and the Perl conversion script: