Performance testing using The Grinder

About a month ago at work, I was trying out a bunch of different performance-testing tools to figure out which one to use to performance-test our software. I ended up discovering a tool called The Grinder which uses Jython to build performance-testing scripts that you can then use to test your application. I even built a little framework in Jython to make the test scripts more modular and reusable. Over the next few days I'll be publishing three articles (this one included) that talk about Grinder.

Introduction

In this article I'll go over installing and setting up Grinder. I am not going to go into too many details since my aim is to provide information that will enable you to have Grinder up and running quickly. If you want more information, you can look at the rather thorough Grinder User Guide. At the end of this guide, you'll know how to install and set up grinder, record a test, modify (and/or parameterize) it, start running it through the console, and record data from the test. Note: The instructions in this guide relate to a Linux environment. You can run Grinder in Windows; the set-up is not much different. The only differences will be in installation locations and shell scripts. If you want more information about setting up Grinder in a Windows environment, please take a look here.

Downloading everything you need

First you need to download Grinder. You can get it from here. You should only need to download the binary distribution and not the source distribution (unless you're planning on extending Grinder). Unzip the installation file to somewhere convenient. I extracted the file to /usr/local/grinder. I recommend chown'ing the installation directory to yourusername:yourusername. Make sure that your version of the JDK is greater than or equal to 1.4, otherwise Grinder won't work! The second thing you need to install is Jython. Note that this is not absolutely necessary, but I found it useful to have around when I was trying to parameterize the test-scripts, and I needed to test something out first. I recommend not using the Jython that comes with Ubuntu. Instead, download and install Jython 2.2.1 from here (there are issues with Jython 2.5 and Grinder).

Setting up Grinder

For demonstration purposes, I'm going to show you how to set up Grinder with a local grinder.properties file, with a local console and a local agent running workers. To get an explanation as to what these terms mean, take a look at the picture below that I shamelessly stole from the Grinder documentation page:

As you can see, you essentially have a console that sits on top of everything. It acts like a controller, pushing out scenarios and test cases (scripts) to agents that are running on load-injection machines. Each agent is in charge of one or more workers that inject load into the system you are testing. The workers also report data back to the console. The console aggregates all the data it receives from the workers and presents it to you. This is the ideal scenario that you would use for actual performance testing, however like I said before, I'm not going to concentrate on that - I'm just going to show you how to quickly set up Grinder and run a performance test on your local machine. If you want to look into distributing scenarios and tests across different agents, you can take a look here.

The first thing you want to do is set up a grinder.properties file. I put this file into the root directory of the checked out app (so basically, wherever you checked out your app). It's not necessary to put it there; you can put it wherever you want. So open up your favorite text editor, and create the following file:

# Tell Grinder how many processes, threads and runs (iterations) we have in this scenario
grinder.processes = 1
grinder.threads = 5
grinder.runs = 10

# grinder.jvm.arguments sends arguments to the JVM. The reason I have the following value
# is to prevent an error (Jython complains about caches) from popping up when you start
# a test. Jython caches imports and so you need to let it know where it can store the imports.
# Though it's not strictly necessary, you do need it if you are performing wildcard imports
# (like java.util.*)
grinder.jvm.arguments = -Dpython.cachedir=/tmp/mycache
grinder.script = grinder.py

# Here you set up the arguments that tells the agent where the console lives.
grinder.hostID = dauntless
grinder.consoleHost = dauntless
grinder.consolePort = 6372
grinder.useConsole = true
grinder.reportToConsole.interval = 500

# sleepTime is a synonym for "think time". You don't want the load-generator hammering away at the
# app one request after another with a uniform interval. sleepTime introduces time variations between
# different requests
grinder.initialSleepTime = 0
grinder.sleepTimeFactor = 1
grinder.sleepTimeVariation = 0.2

# These properties are used to configure the behavior of Grinder with respect to logging
grinder.logProcessStreams = true
grinder.reportTimesToConsole = true
grinder.debug.singleProcess = false
grinder.useNanoTime = false

If you want more information regarding the grinder.properties file, take a look at this page.

Alright, now that you have the grinder.properties created, you also need a few shell-scripts to easily start and stop the console, agent, and TCP Proxy. You will have to change the values of GRINDERPATH, GRINDERPROPERTIES, and JAVA_HOME to values that make sense on your system.

#!/bin/bash
GRINDERPATH=/usr/local/grinder/grinder-3.2
GRINDERPROPERTIES=./grinder.properties
CLASSPATH=$GRINDERPATH/lib/grinder.jar:$CLASSPATH
JAVA_HOME=/usr/local/java/6u13
PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH PATH GRINDERPROPERTIES

#!/bin/bash
. ./setGrinderEnv.sh
java -cp $CLASSPATH net.grinder.TCPProxy -console -http > grinder.py

#!/bin/bash
. ./setGrinderEnv.sh
java -cp $CLASSPATH net.grinder.Console

#!/bin/bash
. ./setGrinderEnv.sh
java -cp $CLASSPATH net.grinder.Grinder $GRINDERPROPERTIES

Creating a Test

Before you start performance testing, you need to have a test. Grinder test scripts are written in the Jython language. You can write your tests from scratch if you wish (it can be useful if you're trying to test things like XML-RPC), but more often than not you will be using Grinder's proxy tool to record a usage scenario and generate a test script. The tool that Grinder comes with is called TCP Proxy. It sits between the browser and the server and records requests and responses that move between the browser and the server. By itself, it simply echoes whatever goes through it (in that sense, it makes a pretty good debugging tool), but we want it to generate Jython test scripts. Note: TCPProxy actually uses XSLT to translate the header information into a Jython script. This is pretty neat, actually because you can use XSLT to translate the code to Groovy, or even JRuby.

First, you need to start up TCP Proxy. Simply run startProxy.sh and you should be good to go. A little window will pop-up letting you know that the proxy is running. There's a textbox in the window that lets you insert comments too. Once you the proxy is started up, you need to configure your browser to use the proxy at localhost:8001. Now access the app you want to test, in your browser. Log in and do whatever it is you need to do (search for a contact, add a contact, send and email, etc.). In the scenario I created, I logged in, searched for contacts, added a new contact, searched for more contacts, and then logged out. Note: If you get an alert box that complains about security certificates, you can add an exception in Firefox by going to Edit->Preferences->Advanced->Encryption Tab->View Certificates->Add Exception. Put in the URL to your server and Firefox should grab the certificate and add an exception.

Once you're done recording the test, press the Stop button in TCPProxy.

WARNING: When you're running TCPProxy, make sure that you have no other tabs open! Also make sure that you have no plugins trying to phone home (through the browser). TCPProxy intercepts every communication that your browser makes so you don't want to pollute your results with irrelevant requests (I think it's possible to filter the data using the domain name, but I haven't looked into how to do it yet)

Parameterizing the test

There are a few tests where you would like to supply data (for example, when adding a contact). In the context of performance-testing, this is called parameterizing. Since Grinder uses Jython, it's actually pretty easy to parameterize a test. First, let's take a look at a TCPProxy-generated Jython-script. I'm only going to reproduce parts of the script since the entire script itself is rather large (however, if you want to look at the entire script, you can download it):

# The Grinder 3.2
# HTTP script recorded by TCPProxy at Jun 17, 2009 4:57:02 PM

from net.grinder.script import Test
from net.grinder.script.Grinder import grinder
from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest
from HTTPClient import NVPair
connectionDefaults = HTTPPluginControl.getConnectionDefaults()
httpUtilities = HTTPPluginControl.getHTTPUtilities()

# To use a proxy server, uncomment the next line and set the host and port.
# connectionDefaults.setProxyServer("localhost", 8001)

# These definitions at the top level of the file are evaluated once,
# when the worker process is started.


connectionDefaults.defaultHeaders = \
  ( NVPair('Accept-Language', 'en-us,en;q=0.5'),
    NVPair('Accept-Charset', 'ISO-8859-1,utf-8;q=0.7,*;q=0.7'),
    NVPair('Accept-Encoding', 'gzip,deflate'),
    NVPair('User-Agent', 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.9.0.11) Gecko/2009060308 Ubuntu/9.04 (jaunty) Firefox/3.0.11'), )

headers0= \
  ( NVPair('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'),
    NVPair('Referer', 'https://local.infusiontest.com:8443/?msg=You%27ve+been+logged+out+-+thanks+for+playing%21&notification=You%27ve+been+logged+out+-+thanks+for+playing%21'), )

headers1= \
  ( NVPair('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'),
    NVPair('Referer', 'https://local.infusiontest.com:8443/Admin/home.jsp'), )

...
...

headers9= \
  ( NVPair('Accept', 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'),
    NVPair('Referer', 'https://local.infusiontest.com:8443/Admin/home.jsp?revertCalUser=true'),
    NVPair('Cache-Control', 'no-cache'), )

headers10= \
  ( NVPair('Accept', 'image/png,image/*;q=0.8,*/*;q=0.5'),
    NVPair('Referer', 'https://local.infusiontest.com:8443/?msg=You%27ve+been+logged+out+-+thanks+for+playing%21&notification=You%27ve+been+logged+out+-+thanks+for+playing%21'), )

url0 = 'https://local.infusiontest.com:8443'

# Create an HTTPRequest for each request, then replace the
# reference to the HTTPRequest with an instrumented version.
# You can access the unadorned instance using request101.__target__.
request101 = HTTPRequest(url=url0, headers=headers0)
request101 = Test(101, 'POST processLogin.jsp').wrap(request101)

request102 = HTTPRequest(url=url0, headers=headers0)
request102 = Test(102, 'GET home.jsp').wrap(request102)

request201 = HTTPRequest(url=url0, headers=headers1)
request201 = Test(201, 'GET popUpTask.jsp').wrap(request201)

...
...

request3004 = HTTPRequest(url=url0, headers=headers0)
request3004 = Test(3004, 'GET defaultLogin.jsp').wrap(request3004)

request3005 = HTTPRequest(url=url0, headers=headers0)
request3005 = Test(3005, 'GET index.jsp').wrap(request3005)


class TestRunner:
  """A TestRunner instance is created for each worker thread."""

  # A method for each recorded page.
  def page1(self):
    """POST processLogin.jsp (requests 101-102)."""

    # Expecting 302 'Moved Temporarily'
    result = request101.POST('/login/processLogin.jsp',
      ( NVPair('username', 'myusername'),
        NVPair('password', 'myp@55W0rd'),
        NVPair('Login', 'Login'), ),
      ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), ))

    grinder.sleep(49)
    request102.GET('/Admin/home.jsp')

    return result

  def page2(self):
    """GET popUpTask.jsp (request 201)."""
    result = request201.GET('/files/popUpTask.jsp' +
      '?0.6697580414936783')

    return result

  def page3(self):
    """POST calendarBackend.jsp (request 301)."""
    result = request301.POST('/Calendar/calendarBackend.jsp',
      ( NVPair('calType', 'Day'),
        NVPair('userId', '1'),
        NVPair('calDate', '17'),
        NVPair('calMonth', '5'),
        NVPair('calYear', '2009'),
        NVPair('weekStartDate', '-1'),
        NVPair('weekStartMonth', '-1'),
        NVPair('weekStartYear', '-1'),
        NVPair('weekEndDate', '-1'),
        NVPair('weekEndMonth', '-1'),
        NVPair('weekEndYear', '-1'), ),
      ( NVPair('Content-Type', 'application/x-www-form-urlencoded; charset=UTF-8'), ))

    return result

  ...
  ...

  def page18(self):
    """POST processContact.jsp (requests 1801-1802)."""

    # Expecting 302 'Moved Temporarily'
    result = request1801.POST('/Contact/processContact.jsp',
      ( NVPair('addingFormId', '0'),
        NVPair('tabs_sel', 'main'),
        NVPair('Contact0Id', '0'),
        NVPair('Contact0FirstName', "Mikey"),
        NVPair('Contact0LastName', "Mike"),
        ...
        ...
        NVPair('Contact0Website', ''),
        NVPair('Contact0Email', "[email protected]"),
        NVPair('email_Contact0Email', '1'),
        ...
        ...
        NVPair('view', 'add'),
        NVPair('Save', 'Save'), ),
      ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), ))
    self.token_view = \
      httpUtilities.valueFromLocationURI('view') # 'edit'
    self.token_tabs_sel = \
      httpUtilities.valueFromLocationURI('tabs_sel') # 'main'
    self.token_msg = \
      httpUtilities.valueFromLocationURI('msg') # 'Person Added'

    grinder.sleep(46)
    request1802.GET('/Contact/manageContact.jsp' +
      '?view=' +
      self.token_view +
      '&tabs_sel=' +
      self.token_tabs_sel +
      '&msg=' +
      self.token_msg)

    return result

  ...
  ...

  def __call__(self):
    """This method is called for every run performed by the worker thread."""
    self.page1()      # POST processLogin.jsp (requests 101-102)

    grinder.sleep(350)
    self.page2()      # GET popUpTask.jsp (request 201)

    grinder.sleep(385)
    self.page3()      # POST calendarBackend.jsp (request 301)

  ...
  ...

    grinder.sleep(407)
    self.page29()     # POST calendarBackend.jsp (request 2901)

    grinder.sleep(3661)
    self.page30()     # GET logout.jsp (requests 3001-3005)


def instrumentMethod(test, method_name, c=TestRunner):
  """Instrument a method with the given Test."""
  unadorned = getattr(c, method_name)
  import new
  method = new.instancemethod(test.wrap(unadorned), None, c)
  setattr(c, method_name, method)

# Replace each method with an instrumented version.
# You can call the unadorned method using self.page1.__target__().
instrumentMethod(Test(100, 'Page 1'), 'page1')
instrumentMethod(Test(200, 'Page 2'), 'page2')
instrumentMethod(Test(300, 'Page 3'), 'page3')
...
...
instrumentMethod(Test(2900, 'Page 29'), 'page29')
instrumentMethod(Test(3000, 'Page 30'), 'page30')

From the above listing, you can get a general idea of the anatomy of a test script. Of course, there's a little bit more that goes into this, so if you want a detailed description, take a look at this page (in my next article, I'll go into more detail about the anatomy of a recorded script). By itself, this script is 95% useful. However, to make it exactly what we need, we need to parameterize certain areas. The first section we need to parameterize is where we add a new contact (look at the page18 method). Right now, it inserts a contact called "Mikey Mike" with the email "[email protected]". That's not particularly useful if we're running this test a bunch of times. We want to parameterize this value. What we're going to do is create a random first name, last name, and email address, and use those values in the request. Here's how we do it (once again, I'm only showing the pertinent sections of the code i.e., the sections that have changed):

# The Grinder 3.2
# HTTP script recorded by TCPProxy at Jun 17, 2009 4:57:02 PM

# We need to import java.util.Random for random number generation
from java.util import Random

...
...

  def page18(self):
    """POST processContact.jsp (requests 1801-1802)."""

    chars = [ "b", "c", "d", "f", "g", "h", "j", "l", "m", "n", "p", "q", "r", "s", "t", "v", "w", "x", "y", "z", "th", "dh", "gh", "ph", "ch", "st", "sw", "sp", "kh" ];
    vowels = [ "a", "e", "i", "o", "u" ];
    r = Random();

    fname = "";
    lname = "";

    for i in range(0, r.nextInt(3) + 2):
       fname += chars[r.nextInt(len(chars))] + vowels[r.nextInt(len(vowels))];

    for i in range(0, r.nextInt(3) + 2):
       lname += chars[r.nextInt(len(chars))] + vowels[r.nextInt(len(vowels))];

    # Expecting 302 'Moved Temporarily'
    result = request1801.POST('/Contact/processContact.jsp',
      ( NVPair('addingFormId', '0'),
        NVPair('tabs_sel', 'main'),
        NVPair('Contact0Id', '0'),
        NVPair('Contact0FirstName', fname.capitalize()),
        NVPair('Contact0LastName', lname.capitalize()),
        ...
        ...
        NVPair('Contact0Website', ''),
        NVPair('Contact0Email', fname + "." + lname + "@" + lname + ".com"),
        NVPair('email_Contact0Email', '1'),
        ...
        ...
        NVPair('Save', 'Save'), ),
      ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), ))
    self.token_view = \
      httpUtilities.valueFromLocationURI('view') # 'edit'
    self.token_tabs_sel = \
      httpUtilities.valueFromLocationURI('tabs_sel') # 'main'
    self.token_msg = \
      httpUtilities.valueFromLocationURI('msg') # 'Person Added'

    grinder.sleep(46)
    request1802.GET('/Contact/manageContact.jsp' +
      '?view=' +
      self.token_view +
      '&tabs_sel=' +
      self.token_tabs_sel +
      '&msg=' +
      self.token_msg)

    return result

As you can see, we've now parameterized the first name and last name values that go into processContact.jsp. There is still one more thing we need to parameterize. If you go back and take a look at grinder.properties, you can see that I set grinder.threads to 5. This means that there are going to be 5 threads running. We don't want the same user logging in more than once (our app doesn't allow it). So what we also need to do is parameterize user logins:

# The Grinder 3.2
# HTTP script recorded by TCPProxy at Jun 17, 2009 4:57:02 PM

from java.util import Random
from net.grinder.script import Test
from net.grinder.script.Grinder import grinder
from net.grinder.plugin.http import HTTPPluginControl, HTTPRequest
from HTTPClient import NVPair
connectionDefaults = HTTPPluginControl.getConnectionDefaults()
httpUtilities = HTTPPluginControl.getHTTPUtilities()

# To use a proxy server, uncomment the next line and set the host and port.
# connectionDefaults.setProxyServer("localhost", 8001)

# These definitions at the top level of the file are evaluated once,
# when the worker process is started.

# Setting up an array of dictionaries (i.e., hash/map/associative-array) of username and password name-value pairs
loginCredentials = [{"username":"vivin", "password":"abAB12!@"},
                    {"username":"jimbo", "password":"abAB12!@"},
                    {"username":"hippy", "password":"abAB12!@"},
                    {"username":"flippy", "password":"abAB12!@"},
                    {"username":"batman", "password":"abAB12!@"}];

...
...

class TestRunner:
  """A TestRunner instance is created for each worker thread."""

  # A method for each recorded page.
  def page1(self):
    """POST processLogin.jsp (requests 101-102)."""

    loginCredential = loginCredentials[grinder.threadNumber];
    username = loginCredential["username"];
    password = loginCredential["password"];
    # Expecting 302 'Moved Temporarily'
    result = request101.POST('/login/processLogin.jsp',
      ( NVPair('username', username),
        NVPair('password', password),
        NVPair('Login', 'Login'), ),
      ( NVPair('Content-Type', 'application/x-www-form-urlencoded'), ))

    grinder.sleep(49)
    request102.GET('/Admin/home.jsp')

    return result

To parameterize logins, I created an array of dictionaries that store username and password combinations. Then, based on the thread number (which is zero-based and which I can access via grinder.threadNumber) I select the appropriate username and password combination, to log in.

Running the tests

Now we can actually start running the tests and recording data! The first thing you need to do is start up your app. Then, you need to start the Grinder console. Use the startConsole.sh shell-script and start up the Grinder console. After that, use startAgent.sh to start up the agent. You should see something like this in your shell:

vivin@dauntless ~/Projects/crm-1.16.0.x
$ startConsole.sh &
[1] 16180

vivin@dauntless ~/Projects/crm-1.16.0.x
$ startAgent.sh
6/18/09 4:43:19 PM (agent): The Grinder 3.2
6/18/09 4:43:19 PM (agent): connected to console at dauntless/127.0.1.1:6372
6/18/09 4:43:19 PM (agent): waiting for console signal

If you see this, it means that your agent started up, and was able to talk to the Grinder console. In the Grinder console, there are four icons in the top-left corner of the window. If you mouse-over the leftmost icon, you will see a tooltip that says "Start the worker processes". Click that button. A dialog box will pop up saying that you haven't selected a properties file and that the worker processes will run the script set in the agent's properties file. This is fine, since you are running Grinder locally and not distributing a file. Click Ok. You should now see something like this in your shell:

vivin@dauntless ~/Projects/crm-1.16.0.x
$ startConsole.sh &
[1] 16180

vivin@dauntless ~/Projects/crm-1.16.0.x
$ startAgent.sh
6/18/09 4:50:12 PM (agent): The Grinder 3.2
6/18/09 4:50:12 PM (agent): connected to console at dauntless/127.0.1.1:6372
6/18/09 4:50:12 PM (agent): waiting for console signal
6/18/09 4:51:48 PM (agent): received a start message
6/18/09 4:51:48 PM (agent): Worker process command line: java '-Dpython.cachedir=/tmp/mycache' -classpath '/usr/local/grinder/grinder-3.2/lib/grinder.jar:' net.grinder.engine.process.WorkerProcessEntryPoint
6/18/09 4:51:48 PM (agent): worker dauntless-0 started
*sys-package-mgr*: processing new jar, '/usr/local/grinder/grinder-3.2/lib/grinder.jar'
*sys-package-mgr*: processing new jar, '/usr/local/java/6u13/jre/lib/resources.jar'
*sys-package-mgr*: processing new jar, '/usr/local/java/6u13/jre/lib/rt.jar'
*sys-package-mgr*: processing new jar, '/usr/local/java/6u13/jre/lib/jsse.jar'
*sys-package-mgr*: processing new jar, '/usr/local/java/6u13/jre/lib/jce.jar'
*sys-package-mgr*: processing new jar, '/usr/local/java/6u13/jre/lib/charsets.jar'
*sys-package-mgr*: processing new jar, '/usr/local/java/6u13/jre/lib/ext/sunpkcs11.jar'
*sys-package-mgr*: processing new jar, '/usr/local/java/6u13/jre/lib/ext/sunjce_provider.jar'
*sys-package-mgr*: processing new jar, '/usr/local/java/6u13/jre/lib/ext/localedata.jar'
*sys-package-mgr*: processing new jar, '/usr/local/java/6u13/jre/lib/ext/dnsns.jar'
*sys-package-mgr*: processing new jar, '/usr/local/grinder/grinder-3.2/lib/jython.jar'
6/18/09 4:51:53 PM (process dauntless-0): starting threads

If you check back into the Grinder console, you can see that it is grabbing data from the agent. Now just sit back and let the test run. Once the test is complete, hit the stop button and exit out of console. If you go to the directory from where you started up the agent, you will see three log files: data_-N.log, error_-N.log, and out_-N.log. data_-N.log is the file that contains all the data gathered (like mean response time, transactions per second, etc.). It is a comma-separated-values file and you can open it up in Excel or Openoffice Calc. error_-N.log contains any errors encountered while running the test, and out_-N.log contains logging data from the script file. You are now in a position to analyze data from the log files. WARNING: I recommend NOT using the version of Java that comes with Ubuntu (GCJ/IcedTea or whatever). Instead, use Sun's Java6. I ran into a whole bunch of problems and wasted an entire day (you can view the thread describing the problem here).

Analyzing Grinder-Generated Data

Grinder spits out raw data. So it's not that user-friendly when compared to, say, WebLoad. This makes sense when you consider the fact that Grinder is geared towards developers. Grinder says "Here is the data I collected, do whatever you want with it". If you want a quick overview of your performance data, there is a tool called GrinderAnalyzer that you can download from here. Usage is pretty straightfoward (check out the documentation). Extract it to your user home directory. Then run the shell script (run.sh) in the bin directory while providing it your data_-N.log and out_-N.log files. GrinderAnalyzer will generate some pretty graphs and some pretty tables for you to view. Your other option is to examine the data on your own and generate graphs. This, while a bit more involved, gives you the freedom to display the data in a manner you see fit.

Conclusion

Grinder is a pretty flexible and powerful performance-testing tool. I'm hoping that after this tutorial you have enough knowledge to go out and start creating and running tests of your own. Like I mentioned before, this guide is by no meand comprehensive. It's simply something to get you started. If you want more information, I highly recommend looking at the excellent Grinder Documentation.

Stay tuned for the next article, where I'll be talking about the anatomy of a recorded Grinder test-script...