This project is now available on GitHub.
Before reading this tutorial (if you haven't already), please took at look at my last two tutorials (Performance Testing using The Grinder and Anatomy of a Grinder test-script). In this tutorial I'll talk about easily writing Grinder test-scripts using a framework I designed. As a disclaimer, I'd like to point out that I'm not a Python programmer and therefore certain things may not be very python-esque. If that's the case, I apologize. My personal opinion is that this framework is especially useful (of course, since I wrote it ;)) for web applications where you have a already have a lot of test data. In that case, you can simply record all your discrete tasks once and then construct different scenarios with them. But if you feel differently and have some constructive criticism, I do look forward to hearing from you! Also, if you'd like to try out the framework I've got a tarball and a zip file available for download on the very last page.
The motivation for a framework
Typically during performance-testing, you want to test different kinds of scenarios. These scenarios are made up of discrete tasks (also known as transactions), for example, consider a scenario where a user logs in, searches for a person, and then logs out. In this scenario, you have three separate tasks: logging in, searching for a person, and logging out. Using Grinder's TCPProxy tool, you can easily record this scenario and you can also parameterize it. What happens when you want to record another scenario? Say, one that involves logging in, searching for a person, adding a new person, and logging out? Sure, you can go ahead and record that, and even parameterize it. But consider the fact that the new scenario is a super-set of the old scenario. It has three tasks in common with the first scenario. What if there was a way to create new scenarios from scratch, not by recording, but by using previously recorded tasks? This way you would only have to record information once, and after that, you can reuse it. To do this, we need to first understand how to identify tasks within a recorded Grinder test-script. Using this as our foundation, we can figure out how to pull out the tasks into discrete units, which we can later reuse.
The Relationship between Requests and Tasks
In the previous tutorial, I went over the anatomy of a recorded Grinder test-script. There, we saw that the script defines a bunch of requests, and then has methods that correspond to each recorded page. In each method, the script executes a bunch of requests. Initially you might think that we would simply pull out these individual methods. However, that is not the case. A single task can involve more than one recorded page. For example, the act of logging into the app involves accessing the login page, then logging in, and then hitting whatever page the login process drops you into. This involves (at the very least) three pages. In fact, the relationship between requests, pages, and tasks looks something like this:
Here, you can see that a set of unique requests belong to a recorded page, and then a set of recorded pages belong to a unique task. Finally, a set of tasks belong to a scenario. What we want to do is pull out the individual tasks so that we can reuse them to create different scenarios. To do this, we will still be using Grinder's TCPProxy tool (at least to record the scenario), but our resulting script will not look like the typical Grinder test-script. It will, instead, conform to the testing framework.
52 thoughts on “Writing Performance Tests in Grinder using a Framework”
Happy onam to you. we are a group of students from cochin who are currently building a web portal on kerala. in which we wish to include a kerala blog roll with links to blogs maintained by malayali’s or blogs on kerala.
you could find our site here: http://enchantingkerala.org
the site is currently being constructed and will be finished by 1st of sep 2009.
we wish to include your blog located here
we’ll also have a feed fetcher which updates the recently updated blogs from among the listed blogs thus generating traffic to your recently posted entries.
If you are interested in listing your site in our blog roll; kindly include a link to our site in your blog in the prescribed format and send us a reply to [email protected] and we’ll add your blog immediatly.
pls use the following format to link to us
hoping to hear from you soon.
I was evaluating Grinder tool..
If the Tester Runner method is big . i mean if Testrunner class exceeds certain length then i am unable to run the test and the tool is throwing following exception…
As a workaround we can divide the script into more than once script but the problem here is how can i call one script in other…?? could you help me..
Thanks In Advance
That’s strange – I’ve never run into that issue before. What is the exception you’re getting? You might consider putting each one into a class of its own and since they’re callable, you might be able to call them split them into separate classes and then write a wrapper that calls them? I think you’ll probably have better luck on the grinder-use mailing list – have you asked your question on there?
I have read some of your posts and find your style of writing very interersting. Salute you on how neatly the articles are presented.
I get following error for all import statements of my framework scripts (ImportError: no module named sci).
This goes away if I compile all my dependent scripts and add jar to classpath of agent. But then I cant parameterize methods in those dependent scripts as they are already classes.
How do I solve this?
I tried settting PYTHONPATH, passing python.home in grinder.properties. nothing seems to work for me.
I found out the issue. I had to put my script sources folder in Jython registry file.@Mukes Nair
Sorry I didn’t get back to you sooner. I didn’t check my email until now! I’m glad that you were able to fix the issue. I wasn’t clear on your initial problem though. Did you have problems importing the actual scripts (that you created using the framework) themselves?
I am trying to use torque framework , facing problem with the scenario class.
As scenario class showing error at following code.
” urlDict = property(getUrlDict, setUrlDict)” because my eclipse is unable to find property import.
i have jython 2.1 with me. it seems the property import is not available on the jython 2.1
Could you please help me wat might be the problem..?
I need a a way to share some data between tasks..
How can i do this?
Do we need to add other prop in the scenario class
i am able to fix this error.. the problem is with my environment
Actually i am using GrinderStone its eclipse plugin for Grinder Tool.
Sorry I wasn’t able to get to you sooner! Glad you were able to fix the issue – it did sound like an environment problem.
What data are you trying to share between tasks?Tasks are discrete entities that should have nothing in common with each other. Hence, they shouldn’t be sharing state-data. However, they can share input data (from a common resource – like a file).
suppose consider following usecase:
1.login is one task: after user got logged in we will get a userid .
2.second task :
— Here the request in this task need the userid from the previous task.
So, we need a global parameters that can be accessible throughout the scenario.
Could you suggest me how we can achieve . I have one approach in my mind.
— Add another param like ‘urlDict’ in the Scenario Class . what do you say.?
In this case I’d parameterize the usernames and userId’s and link them to the thread number. You’d have to do this even if you were testing a pure Grinder-script that didn’t use the framework.
I have another issue to resolve. Any help will be appreciated. Lets take a scenario of a billing counter like of BestBuy. The associate will log in and load the billing screen. Then he will enter items and print receipt for a customer. Then for the next customer and so on. Here I have classified tasks in the scenario as follows
1. Login Task
2. Load Billing Screen Task
3. Enter Items Task
4.Print receipt Task
5. Logout task
Now in the run I want to run tasks 1, 2 and 5 only once. I want to repeat 3 and 4.
How do I do this?
I’m assuming that you know how many times you need to repeat tasks 3 and 4? You can simply create n “Enter Items” tasks and n “Print receipt” tasks (through a loop perhaps, and maybe have an array for each, for example enterItemsTaskArray and printReceiptTaskArray).
When you build your scenario, you add Task 1 and 5, then iterate through both arrays at the same time and them to the scenario (so you’d add one “Enter items” task and then one “Print receipt” task). After you’re done you add Task 5.
Actually I don’t know how many times I will be repeating 3 and 4. I found another way to tackle the issue. Like this.
log(“initializing scenario %s” % self.__class__.__name__)
log(“running scenario %s” % self.__class__.__name__)
log(“finalizing scenario %s” % self.__class__.__name__)
so my scenario will look like:
I think this should solve my current issue.
That would work as well. I’m assuming that you’re using meta-programming and extending the Scenario class at runtime?
Even if you didn’t know how many times you’d be repeating it, as in let’s say that it will be repeated a random number of times, you can add a random number of items to the arrays. But I do like your solution. It’s a little more elegant.
I have a problem of data getting mixed up between threads. Look at the log below.
initially thread-1 had value 100000019. Once thread -0 got the value 100000024, i printed out values again. Now both threads have the same value.
I decompiled one of the class files and I found that self is declared as:
static final py self;
Did you face similar issue?
11/20/09 10:56:09 AM (thread 1 run 0 test 101): self.parameters[“SelectStationAndScanShipment2”][“201”][“BarcodeData”] 100000019
11/20/09 10:56:09 AM (thread 0 run 0 test 101): suggestedShipments [‘100000019’, ‘100000024’]
11/20/09 10:56:09 AM (thread 0 run 0 test 101): self.parameters[“SelectStationAndScanShipment2”][“201”][“BarcodeData”] 100000024
11/20/09 10:56:09 AM (thread 1 run 0 test 4201): http://10.10.20.80:7099/smcfs/console/exuipack.exui -> 200 OK, 365 bytes
11/20/09 10:56:09 AM (thread 1 run 0 test 4201): sleeping for 29 ms
11/20/09 10:56:09 AM (thread 0 run 0 test 4201): http://10.10.20.80:7099/smcfs/console/exuipack.exui -> 200 OK, 919 bytes
11/20/09 10:56:09 AM (thread 0 run 0 test 4201): sleeping for 31 ms
11/20/09 10:56:09 AM (thread 1 run 0 test 4301): http://10.10.20.80:7099/smcfs/console/exuipack.exui -> 200 OK, 244 bytes
11/20/09 10:56:09 AM (thread 1 run 0 test 4301): sleeping for 34 ms
11/20/09 10:56:09 AM (thread 0 run 0 test 4301): http://10.10.20.80:7099/smcfs/console/exuipack.exui -> 200 OK, 244 bytes
11/20/09 10:56:09 AM (thread 0 run 0 test 4301): sleeping for 30 ms
11/20/09 10:56:09 AM (thread 1 run 0 test 4401): http://10.10.20.80:7099/smcfs/console/exuipack.exui -> 200 OK, 919 bytes
11/20/09 10:56:09 AM (thread 0 run 0 test 4401): http://10.10.20.80:7099/smcfs/console/exuipack.exui -> 200 OK, 919 bytes
11/20/09 10:56:09 AM (thread 1 run 0 test 4401): sleeping for 63 ms
11/20/09 10:56:09 AM (thread 0 run 0 test 4401): sleeping for 60 ms
11/20/09 10:56:09 AM (thread 1 run 0 test 4501): http://10.10.20.80:7099/smcfs/console/exuipack.exui -> 200 OK, 341 bytes
11/20/09 10:56:09 AM (thread 1 run 0 test 4600): self.parameters[“SelectStationAndScanShipment2”][“201”][“BarcodeData”] 100000024
11/20/09 10:56:09 AM (thread 0 run 0 test 4501): http://10.10.20.80:7099/smcfs/console/exuipack.exui -> 200 OK, 341 bytes
11/20/09 10:56:09 AM (thread 0 run 0 test 4600): self.parameters[“SelectStationAndScanShipment2”][“201”][“BarcodeData”] 100000024
How are you parameterizing the data? Are you doing it based on the thread number?
am setting data in the task itself using one method call to a utility script, like below
“””Selecting station and scanning shipment POST exuipack.exui (request 201).”””
self.parameters[“SelectStationAndScanShipment2”][“201”][“BarcodeData”] = self.StringUtil.getAPickedShipmentNumber(self.parameters[“SelectStationAndScanShipment2”][“201”][“OrganizationCode”])
self.log(“self.parameters[\”SelectStationAndScanShipment2\”][\”201\”][\”BarcodeData\”] %s” % self.parameters[“SelectStationAndScanShipment2”][“201”][“BarcodeData”])
I recommend not modifying the actual task. What you want to do is create a parameterizing method that sets the values in the parameters dictionary for you. Check out my examples in this post.
i wanna show you a grinder xml which is not producing correct url in the task created by the perl script. I directly created grinder script using proxy. I can see a lot of arguements witht he url. But when it is converted using using the perl script into a task, all the arguements are missing. let me know where to send the related files so that you can take a look. I haven’t looked into perl yet. So I thought you could resolve it faster.
Is it a POST or a GET? I can take a look at it. I DM’ed you my email on twitter. So just send me the XML and I’ll take a look at it.
Here i need your help. I am working in web application and i wanna to test the load of the application. For this i am using the grinder tool and under stand all the things except properties file of grinder framework. Is it possible to run more than one script or divide the scripts in threads on same time. Here i can explain you through example. Suppose we have 4 scripts and want 4 scripts are equally divided in 100 user’s. is it possible?
I looked around for ways to run multiple scripts and I ran into this post here that says you can do it by making each script as a separate property and not space delimited. So what you need to do is divide the load (and select a user) based on the thread number. Script 1 will use user1-user25 (if the thread number is between 1 and 25), script 2 will use user26-user50 (if the thread number is between 26 and 50). This basically means that you need to have 100 threads.
The Perl file under the link on page 9 is not the same as the one you packaged in the ZIP.
Thanks for pointing that out Marcel; I’ll fix it!
Could you show the details about it. I have the same problem.
I have a question about the using of regex. Since grinder just allows to compile regex within the runner, how do i parse http results in torqueo tasks? I didn’t get any valuable answers in the internet so far. Do you have any advices? Would be great, thx!
Sorry for taking so long to respond. I was pretty busy with school and work. There is no way (as of now) to have access to the HTTP results. It’s abstracted away inside the tasks. What you could do, is modify the autogenerated task-files themselves and get access to the result that way.
The reason that the result is not provided (and possibly cannot be provided per the current design) is that a task consists of one or more requests. So it doesn’t make sense to talk about the “result” of a task. A task represents an abstract unit.
Thanks for your answer. I actually meant a http request only and was not able to parse it. The problem was, that it’s hardly (or even not?) possible to use the python regex lib in jython. I’m now just using the java regex library and it works like a charm.
Thanks anyway for your answer! You did a great job with torqueo, it’s a great tool which helped me a lot.
Glad to know you figured it out. Also, thank you for your kind words 🙂 I’m glad you find my tool helpful!
Thanks for describing the test framework. Its nice to see the way u maintained the tutorials for Grinder. Have few queries on implementing few features related to Grinder.
1. How to call a user defined function
2. How to monitor server side metrics like CPU,memory and Disk etc.
3. Do we have an option to add users dynamically.
Do u have any monitoring template available
Great work! I was looking at designing a framework for grinder myself before Is tumbled into your article, I like your framwork a lot.
btw. the link to the example http://vivin.net/wp-content/uploads/2009/09/grinder-frameworked.py is unavailable atm.
@wauter: Hello Wauter, I’m glad you liked the framework! Looks like I lost the content when I migrated my content over. Here’s a link to the project on Github. I’ll update the post!
@Sattish: What do you mean by a user-defined function? As far as hardware metrics, grinder might provide the functionality – take a look at their documentation. Also, what do you mean by “adding users dynamically”?
@wauter: Could you please provide me active link to grinder-frameworked.py because i can`t find them in git.
Awesome work! I think that your work illustrates how flexible Grinder really is. I’m using it myself to parse JSON and XML from a number of web service calls and it works great ;). I also wanted to take a look at the grinder-frameworked.py module and was wondering if you had a chance to locate the file. Thanks!
@Bobby: I’ve updated the link on this page. You can find the file here. I don’t know why, but there is some weird redirection rule somewhere that prevents me from linking to .py files.
@Bill: I’ve updated the link! Glad you found the tutorial useful!
Hi Vivin, I have no questions as I’m only about to try to implement your framework now but I just wanted to say thanks for making this available to everyone and for supporting it on here like you do, great work!
I’m using your framework in eclipse\grinderstone\windows environment to performance test as well as replaying logs that I cull from splunk.
Hi Vivin, That you for the framework I was looking for a way to reuse my tests and I loved your idea.
Unfortunately I follow your examples and even copy and execute them, but I have always the same mixed up result.
For example running 11 threads, one process and one run gives this output if I print self.parameters in appLogin2
“””Log into the app GET home.jsp (requests 201-202).”””
print ‘thread=%s Login=%s password=%s username=%s’ % (grinder.threadNumber, self.parameters[“appLogin2”][“201”][“Login”], self.parameters[“appLogin2”][“201”][“password”],self.parameters[“appLogin2”][“201”][“username”])
thread=5 Login=Login password=abAB12!@ username=hippy
thread=9 Login=Login password=abAB12!@ username=hippy
thread=3 Login=Login password=abAB12!@ username=hippy
thread=10 Login=Login password=abAB12!@ username=hippy
thread=4 Login=Login password=abAB12!@ username=hippy
thread=8 Login=Login password=abAB12!@ username=hippy
thread=6 Login=Login password=abAB12!@ username=hippy
thread=0 Login=Login password=abAB12!@ username=hippy
thread=1 Login=Login password=abAB12!@ username=hippy
thread=7 Login=Login password=abAB12!@ username=hippy
thread=2 Login=Login password=abAB12!@ username=hippy
As you can see the same user will appear in different threads. I suppose that is happening because self.parameters are global and not local to TestRunner.
I think that is not not suppose to happen.
Can you tell me if I am doing something wrong ?
To have this strange result the only thing that I have to do is to put the ginder.sleep(50) in appLogin2.
Without sleep I have:
thread=2 Login=Login password=abAB12!@ username=hippy
thread=10 Login=Login password=abAB12!@ username=will
thread=1 Login=Login password=abAB12!@ username=jimbo
thread=6 Login=Login password=abAB12!@ username=worf
thread=7 Login=Login password=abAB12!@ username=deanna
thread=5 Login=Login password=abAB12!@ username=beverly
thread=3 Login=Login password=abAB12!@ username=flippy
thread=8 Login=Login password=abAB12!@ username=jean-luc
thread=0 Login=Login password=abAB12!@ username=vivin
thread=4 Login=Login password=abAB12!@ username=batman
thread=9 Login=Login password=abAB12!@ username=laren
@Miguel Lamy: Hello Miguel, I’m glad you found the framework useful. It seems interesting that this happens only if you put the grinder.sleep(50). Is there a reason that you need the sleep in there? I wrote this framework a few years ago and I wonder if changes to Grinder are causing this to happen.
That being said, it picks up the credentials based on the thread number so I wonder why this is happening. I would add a print statement right after it sets the password to see what the values are (in parameterizeLogin). I wonder if in between that statement and the time it’s being used, another thread is overwriting the value. It’s been a while since I wrote this so I’m not sure if I considered thread-safety as far as self.parameters is concerned. But I’d start looking there. Unfortunately, I’m pretty busy these days with my Masters, work and other side projects and so I don’t have very much time to look into this. However, do let me know what you’re able to find. I might be able to point you in the right direction or suggest a fix.
Hello Vivin, Thank you for your quick reply.
I did a few experiments before and I’m almost pretty sure that framework it’s not thread safe.If I print the values in parameterizeLogin (defined only in the scenario) the values are always OK. I only use sleep for test purposes when I use requests instead of sleep I have the same results. Most of the time I have different repeated values in different threads (and not always the same result in all threads).
I also tried to print values in run method in the logintask and the print shows that self.parameters values are not ok after the call self.appLogin2() but self.parameters are ok before that call.
“””The run() method runs all the tests in this Task”””
print here show self.parameters values are OK
print here show self.parameter values are not OK
Can you suggest a fix or direction based on this ?
I’m using grinder 3.11
@Miguel Lamy: No problem! Glad to help! I’m not extremely familiar with Python/Jython. Are there any synchronization primitives? Although in this case even with synchronization, there is nothing preventing one thread from from resetting the value after one thread has finished. A fix will be a little more involved, I think. My first (by no means most practical, or efficient) would be to segment the data based on thread number. Perhaps a data-structure that’s local to the thread, or a global data structure that is keyed by thread number. So in that case self.parameters could have another key so that when you’re accessing the data, you can basically do self.parameters[grinder.threadNumber][“appLogin2”][“201”][“username”]. That’s the direction that I’d go.
Hello again Vivin 🙂
My first thought when I see the thread issue was to parameterize data the way you describe – that is something like self.parameters[grinder.threadNumber][“appLogin2”][“201”][“username”]. But I’m afraid that kind of solution will be very inefficient and will use lots of memory, so I didn’t implement that solution yet.
I was looking for a way to implement a data structure that will be local to the thread, but in that case I think I’ll need have to have all my method calls defined in run() in TestRunner in the scenario.
Do you agree with me or am I missing something ?
@Miguel Lamy: Hello Miguel 🙂
Yes, that is definitely a memory-inefficient solution because it will grow linearly according to the number of threads that are running. I took a look at the Grinder documentation and it appears that a new instance of TestRunner is created for each thread and so it can be used to store thread-specific data (see here; just search for “thread-specific”). So I imagine that you might be able to communicate that information to the currently running TestRunner instance? I’d explore along those lines. 🙂