Codespeed is a project for tracking performance. I discovered it when the PyPy project started using Codespeed to track performance. Since then development has been done to make its setup easier and provide more display options.

Anyway, two posts ago I talked about running nightly benchmarks with Hudson. Then in the previous post I discussed passing parameters between builds in Hudson. Both of these posts are worth reading before trying to setup Hudson with Codespeed.

Codespeed Installation/Configuration

Django Quickstart

Codespeed is built on Python and Django. Some basic knowledge of Django is needed in order to get everything up and running. Don’t worry, it’s not that hard to learn the bit that is needed. manage.py is all you need to know about to setup and view Codespeed. There is information about deploying Django to a real web server, but I won’t be covering that here.

Here are the commands to get Django running:

syncdb

syncdb is used to initialize the database with the necessary tables. It will also setup an admin account. With the sqlite3 database selected, it will create the database file when this command is run.

The command is:

python manage.py syncdb

runserver

The next command is the runserver command. This runs the built-in django server. In the documentation they state you’re not supposed to use it in a production environment, so make sure to deploy to a production environment if you plan to host it on the Internet or high traffic network.

The command is:

python manage.py runserver 0.0.0.0:9000

By default the server will run on 127.0.0.1:8000. Setting the IP to 0.0.0.0 allows connections from any computer. This works well if you’re on a local area network and want to set it up on a VM over SSH, but still be able to access the web interface from your computer. The port is the port for the server to run on. To view Codespeed, point your browser at 127.0.0.1:9000 or the IP of the machine it’s on with the colon 9000.

Django has many settings that may or may not need to be tweaked for your environment. They can be set through the speedcenter/settings.py file.

Codespeed Setup/Settings

Now for setting up the actual Codespeed server. First check it out using git. The clone command is:

git clone http://github.com/tobami/codespeed.git

The settings file is speedcenter/codespeed/settings.py.

Most of the default values will work fine. They’re mostly for setting default values for various things in the interface.

One thing that does need to be configured is the environment. Start by running the syncdb command and then run the server using runserver. Now that the server is running, browse to the admin interface. If you ran the server on port 9000, point your browser at http://127.0.0.1:9000/admin. Login using the username and password you created during the syncdb call. A Codespeed environment must be created manually. The environment is the machine you’re running the benchmarks on. After logging in, click Add next to the Environment label. Fill in the various fields and remember the name of it. Save it when you’re done. The name will be used later when submitting benchmark data to Codespeed.

Submitting Benchmarks

This will pick up where my last tutorial left off. The benchmarks were running as a nightly job in Hudson. Sending benchmark data to Codespeed will take a bit of programming. I’m going to continue the example with JRuby, so the benchmarks and submission process are written in Ruby.

In order to submit benchmarks information must be transferred from the JRuby build job to the Ruby Benchmarks job. My last post discussed how to transfer parameters between jobs. Using the Parameterized Trigger Plugin and passing extra parameters using a properties file will allow you to get all the necessary parameters to the benchmarks job.

The required information for submitting a benchmark result to Codespeed includes:

• commitid – The id of the commit, which could either be a git/mercurial hashcode or an svn revision number.
• project – The name of the project to save.
• executable – The name of the executable.
• benchmark – The name of the benchmark.
• environment – This is the name of the environment you created earlier. It must be the name of an existing environment.
• result_value – The runtime of the benchmark. You can configure what units a benchmark has through the admin interface. Default is seconds.

This information can be included but is optional:

• std_dev – The standard deviation of the results of the benchmarks.
• min
• max
• branch – The branch corresponding to this benchmark in the SCM repository.
• result_date – The timestamp of the commit in the form “%Y-%m-%d %H:%M”

The above information is passed to Codespeed through an encoded URL. Have the URL point to http://127.0.0.1:9000/results/add/ and encode the parameters for sending. For the JRuby benchmarks, the following parameters are sent from the JRuby job to the to the ruby benchmarks job.

COMMIT_ID=$(git rev-parse HEAD)
COMMIT_TIME=$(git log -1 --pretty=\"format:%ad\")
RUBY_PATH=$WORKSPACE/bin/jruby
REPO_URL=git://github.com/jruby/jruby.git

The other fields are derived from the benchmarks job itself.

Here is the source code for submission through Ruby:

output = {}
canonical_name = doc["name"].gsub '//', '/'
output['commitid'] = commitid
output['project'] = BASE_VM
output['branch'] = branch
output['executable'] = BASE_VM
output['benchmark'] = File.basename(canonical_name)
output['environment'] = environment
output['result_value'] = doc["mean"]
output['std_dev'] = doc["standard_deviation"]
output['result_date'] = commit_time
 
res = Net::HTTP.post_form(URI.parse("#{server}/result/add/"), output)
puts res.body

It’s a good idea to always print out the response as it will contain debug information. There is an example of how to submit benchmarks to Codespeed using Python in the Codespeed repository in the tools directory.

Viewing Results

After results are in the the Codespeed database, you can view the data through the web interface. Direct a browser at http://127.0.0.1:9000. The changes view shows the trend over the last revisions. The timeline view allows you to see a graph of recent revisions, and the newly added comparison view will compare different executables running the same benchmark.

I decided to give Hudson a try because I had heard good things about it.  Also after hearing coworkers complain about cruise control and cdash, I thought I’d try something new.  Since Hudson has pretty extensive documentation, I’ll walk you through setting up the JRuby project to build with Hudson and getting benchmarks running on it.

Hudson Installation

On Ubuntu it’s as simple as:

sudo apt-get install hudson

While I didn’t install it on windows, the installation should require little more than installing Tomcat and then downloading the Hudson war file and put it in the web-apps directory.

After installation browsing to http://127.0.0.1:8080 should show the Hudson Dashboard.

Hudson Configuration

After Hudson installation is complete, it requires very little configuration before setting up your first project.  One thing that may be necessary is going to the plugins page and making sure your version control system is covered.  For setting up a continuous integration machine to build JRuby, the git plugin is necessary.

To install the Hudson Git Plugin, click Manage Hudson on the left hand side.  Then click Manage Plugins from the list in the middle of the screen.  Click the Available tab, and find the Hudson GIT plugin in the list.  After it’s installed it will show up in the Installed tab.

After installing all the necessary plugins for your project go back to the Hudson Dashboard by clicking the Hudson logo, or the Back to Dashboard link.

Setting up a Project to Build

A good first step it to make sure the project will build on the given machine without being built through Hudson.  There may be some dependencies that got overlooked, and this is a good way to make sure everything is setup to build your project.

Now, click on the New Job link on the left hand side.  For the JRuby project, the Build a free-style software project is the type of project to setup.  I imagine that is the correct type of project to setup for most projects.

Unless you plan on keeping all the builds produced on the server, the Discard Old Builds is a good option to check, and set how long you want the builds to remain on the server.  Choose the source code management tool that you use for your project, which is Git for JRuby, and set the appropriate settings.

JRuby settings:

URL of Repository: git://github.com/jruby/jruby.git
Branch Specifier (blank for default): master
Repository browser (Auto)

There are several types of Build Triggers by default.  More Build Triggers can be added through plugins, if you’re looking for another way to trigger a build.  For a nightly build at midnight select the Build periodically option, and put @midnight in the field.

For the build step, if you’re building a Java project select Invoke Ant.  Otherwise, Execute shell may be a good option for you.  For JRuby, select Invoke Ant and set the target to jar to build it.

At this point you can click the Save button at the bottom of the page and click Build Now on the next page to build your project.  It’s a good idea to make sure your project builds correctly before trying to add in nightly benchmarks.  It’s easier to debug problems before you have too much going on.  By clicking on the build from the active builds list the console output can be seen from the browser.

Running the Benchmarks

If your benchmarks are in the same repository, you’re mostly done.  Add another build step, and set it up to run your benchmarks.  While JRuby does have benchmarks in its repository, the benchmarks I plan on running are in a different repository.  With this goal in mind, I created another Job in Hudson to checkout and run the benchmarks.

Its setup is very similar to that of JRuby, it checks out the source and runs the benchmarks.  The main difference is that a parameter is passed to the project to tell it which Ruby VM to use.  The Parameterized Trigger Plugin is necessary to pass a parameter from one project to another.  The way it works is you set a parameter in the project receiving the parameter near the top of the page.  In my case, I added a RUBY_PATH parameter.  Then you setup the build job to send that parameter to the benchmarks job.

To do this, I went back to the JRuby job and turned on the Trigger parameterized build on other projects option.  It should be the last option down at the bottom of the page.  I set the JRuby job to trigger with the benchmarks job name, and in the predefined parameters field I put the following:

 RUBY_PATH=$WORKSPACE/bin/jruby

After this is in place, when a JRuby build finishes it will start a benchmarks run.  Now that your benchmarks are up and running, the next part to this series will go over how to display the information in a way that makes it easy to spot regressions.

If you have any questions or if I went over something too quickly, post a comment and/or ask a question.

Anyway, when I was trying to use the Netbeans debugger on JRuby it wasn’t stopping at my breakpoints, and I couldn’t figure out why.  I thought it might have something to do with me using linux, and after people ran out of ideas they seemed to think the same thing.  This turned out to not be the case.  It was a problem with the ant script for running the debugger.

It was as simple as changing this:

    <nbjpdastart addressproperty="jpda.address" name="JRuby" transport="dt_socket">
      <classpath>        
        <pathelement path="${jruby.classes.dir}"/>
        <path refid="build.classpath"/>
      </classpath>      
    </nbjpdastart>

to this:

    <nbjpdastart addressproperty="jpda.address" name="JRuby" transport="dt_socket">
      <classpath>        
        <pathelement path="${jruby.classes.dir}"/>
        <path refid="build.classpath"/>
      </classpath>      
      <sourcepath path="${src.dir}"/>
    </nbjpdastart>

The sourcepath makes all the difference. I’m posting this because nbjpdastart has very little documentation, and it took me quite a while to figure this out. I hope this saves you some time.

I started by looking around on wikipedia, to see what that had to offer.  IntroSort caught my eye (also see: A guide to Introsort).  I thought it was interesting that after recursing to a certain depth it would switch from quicksort to heapsort.  I don’t think this optimization turned out to be that needed in the end though.  Other than switching to heapsort, it was pretty much a median of three quicksort with an insertion sort added.

It was a good starting point though.  I switched the heapsort for a shell sort and that dropped the number of comparisons needed by a good amount.  One thing I saw was only my “Median of 3 Killer” test case was affected by that.  I searched around the Internet often during this, and came across this page: QuickSort.  It had some interesting ideas like grouping the same element together when running the partition function.  I tried implementing that several times and it always ended up increasing the number of comparisons and the runtime.  I’m not 100% sure why.  If someone who knows more about sorting that I do has any input on that, I’d be happy to hear it.

Anyway, after working on that for a while I took a look at the competition.  I looked at Ruby’s C code.  They do some interesting things, which I hadn’t thought of up to that point.  First, they take the median of 7 if it has more than 200 elements.  Second, they don’t sample the end elements, this helps me out later in my optimizations.  I didn’t copy exactly what they were doing, but did a similar idea.  One thing they also did was look at the order of the 3 values that they compared last (I’ll refer to these as v1, v2, and v3).  v1 is before v2 and v3 in the list’s current state.  v2 is in the middle of the other two, and so on.

They had a lot more checks than what I ended up using, but I check if v1 <= v2 <= v3.  If this is the case I run the sequential test.  I also check if v1 >= v2 >= v3 to see if the list is in reversed order, and if it is I reverse the list before continuing.  While running these tests, I don’t check the first or last element of list because I have a separate test for that.  If it passes the sequential or reverse test that means the entire list is sorted except for potentially the first and/or the last element.  I then do a test on them and if they’re out of sequence I do bubble sort style swaps until they’re in the correct location.

Checking the end was one of the last optimizations I performed.  The main reason I added it is that case can be slow with the normal sorting algorithm, and I don’t think it’s that uncommon a case.  I’ve seen it happen where an element is appended to a sorted list and then the list is sorted again.  Overall, this catches a case with the potential to be slow in a fairly cheap manner.  These cases weren’t weeded out by the v1 <= v2 <= v3 style checks because the median of 7 that I use doesn’t check the end elements.

One of the last optimizations I performed was converting it to use a stack rather than operating recursively.  To be honest, this provided more speedup than I was expecting.  I guess all the function calls were taking a toll on the performance that I just didn’t realize at the time.

Another implementation technique that I ran tests to figure out which was better was whether to do insertion sort at the end on the entire list, or to do it as I find sections that are smaller than the threshold value.  After running benchmarks it turned out that putting it at the end was better.  I suspect that the extra function calls required for it to be done during the quicksort loop was more overhead than the potential gain of having more cache locality.

On with the benchmarks: (The numbers are time in seconds.  In parenthesis is speedup over Java.)

                          Java                 My Qsort
1245.repeat.1000.txt      1.6552300e-05 (1.00) 8.1874300e-06   (2.02 )
1245.repeat.10000.txt     0.00026284956 (1.00) 0.00012648017   (2.08 )
end.0.1000.txt            5.8904900e-06 (1.00) 1.7295600e-06   (3.41 )
end.0.10000.txt           8.3629030e-05 (1.00) 2.3148840e-05   (3.61 )
identical.1000.txt        3.5435500e-06 (1.00) 5.9168000e-07   (5.99 )
identical.10000.txt       3.7097050e-05 (1.00) 6.5003600e-06   (5.71 )
med.3.killer.1000.txt     1.0182050e-05 (1.00) 7.2132600e-06   (1.41 )
med.3.killer.10000.txt    0.00013968449 (1.00) 9.4941470e-05   (1.47 )
rand.dups.100.txt         1.2044400e-06 (1.00) 6.3194000e-07   (1.91 )
rand.dups.1000.txt        1.9847760e-05 (1.00) 1.0417630e-05   (1.91 )
rand.dups.10000.txt       0.00031415178 (1.00) 0.00020365385   (1.54 )
rand.no.dups.100.txt      1.2328200e-06 (1.00) 8.1612000e-07   (1.51 )
rand.no.dups.1000.txt     1.9309830e-05 (1.00) 1.1890280e-05   (1.62 )
rand.no.dups.10000.txt    0.00027436851 (1.00) 0.00017424722   (1.57 )
rand.steps.1000.txt       1.6057600e-05 (1.00) 1.0023700e-05   (1.60 )
rand.steps.10000.txt      0.00019955369 (1.00) 0.00017004971   (1.17 )
rev.ends.1000.txt         1.2306600e-05 (1.00) 2.8749300e-06   (4.28 )
rev.ends.10000.txt        9.4499880e-05 (1.00) 2.7255840e-05   (3.47 )
rev.partial.1000.txt      1.7107210e-05 (1.00) 8.7564000e-06   (1.95 )
rev.partial.10000.txt     0.00024198949 (1.00) 0.00013045623   (1.85 )
rev.saw.1000.txt          1.6793840e-05 (1.00) 9.4294600e-06   (1.78 )
rev.saw.10000.txt         0.00025133096 (1.00) 0.00014296088   (1.76 )
reverse.1000.txt          1.4600270e-05 (1.00) 1.1565800e-06   (12.6 )
reverse.10000.txt         0.00020535965 (1.00) 1.3798890e-05   (14.9 )
seq.0.is.1000.1000.txt    6.5672500e-06 (1.00) 1.4837800e-06   (4.43 )
seq.0.is.1000.10000.txt   4.7335130e-05 (1.00) 7.2842900e-06   (6.50 )
seq.partial.1000.txt      1.5497830e-05 (1.00) 6.0515300e-06   (2.56 )
seq.partial.10000.txt     0.00022936435 (1.00) 9.1791120e-05   (2.50 )
seq.saw.1000.txt          1.1645670e-05 (1.00) 4.6621150e-05   (0.250)
seq.saw.10000.txt         0.00019771144 (1.00) 0.00011184647   (1.77 )
sequential.1000.txt       3.4216700e-06 (1.00) 5.8689000e-07   (5.83 )
sequential.10000.txt      3.7812430e-05 (1.00) 6.1648500e-06   (6.13 )

These benchmarks were taken by running the sorting algorithm on each dataset 10000 times (to warm up the JVM), and then running it on the data 10 times to time it.  I took the average of those 10 runs.  The only case where My Qsort was slower than the built-in Java Arrays.sort() was seq.saw.1000.txt.  I attribute this to noise.  I ran it again and got the following:

seq.saw.1000.txt          0.00013980571 (1.00) 0.00012667049   (1.10 )

Hopefully this makes JRuby’s sorting comparable to Ruby’s.  One note, while Java’s sort is stable, the quicksort I wrote is not.  All that means is that if multiple entries have the same value they may get sorted differently.

You can do whatever you’d like with the source code.  If you do find it useful some credit and/or a link to here would be nice.

Here are the test files: Test Files

Here is the sort itself: Sorting Algorithm