Nightly Benchmarks: Setting up Hudson

For some projects, finding out about performance regressions is important.  I’m going to write a two part series about setting up a nightly build machine and displaying the generated data.  This part is going to cover installation of Hudson, and getting the benchmarks running nightly.

I decided to give Hudson a try because I had heard good things about it.  Also after hearing coworkers complain about cruise control and cdash, I thought I’d try something new.  Since Hudson has pretty extensive documentation, I’ll walk you through setting up the JRuby project to build with Hudson and getting benchmarks running on it.

Hudson Installation

On Ubuntu it’s as simple as:

sudo apt-get install hudson

While I didn’t install it on windows, the installation should require little more than installing Tomcat and then downloading the Hudson war file and put it in the web-apps directory.

After installation browsing to http://127.0.0.1:8080 should show the Hudson Dashboard.

Hudson Configuration

After Hudson installation is complete, it requires very little configuration before setting up your first project.  One thing that may be necessary is going to the plugins page and making sure your version control system is covered.  For setting up a continuous integration machine to build JRuby, the git plugin is necessary.

To install the Hudson Git Plugin, click Manage Hudson on the left hand side.  Then click Manage Plugins from the list in the middle of the screen.  Click the Available tab, and find the Hudson GIT plugin in the list.  After it’s installed it will show up in the Installed tab.

After installing all the necessary plugins for your project go back to the Hudson Dashboard by clicking the Hudson logo, or the Back to Dashboard link.

Setting up a Project to Build

A good first step it to make sure the project will build on the given machine without being built through Hudson.  There may be some dependencies that got overlooked, and this is a good way to make sure everything is setup to build your project.

Now, click on the New Job link on the left hand side.  For the JRuby project, the Build a free-style software project is the type of project to setup.  I imagine that is the correct type of project to setup for most projects.

Unless you plan on keeping all the builds produced on the server, the Discard Old Builds is a good option to check, and set how long you want the builds to remain on the server.  Choose the source code management tool that you use for your project, which is Git for JRuby, and set the appropriate settings.

JRuby settings:

URL of Repository: git://github.com/jruby/jruby.git
Branch Specifier (blank for default): master
Repository browser (Auto)

There are several types of Build Triggers by default.  More Build Triggers can be added through plugins, if you’re looking for another way to trigger a build.  For a nightly build at midnight select the Build periodically option, and put @midnight in the field.

For the build step, if you’re building a Java project select Invoke Ant.  Otherwise, Execute shell may be a good option for you.  For JRuby, select Invoke Ant and set the target to jar to build it.

At this point you can click the Save button at the bottom of the page and click Build Now on the next page to build your project.  It’s a good idea to make sure your project builds correctly before trying to add in nightly benchmarks.  It’s easier to debug problems before you have too much going on.  By clicking on the build from the active builds list the console output can be seen from the browser.

Running the Benchmarks

If your benchmarks are in the same repository, you’re mostly done.  Add another build step, and set it up to run your benchmarks.  While JRuby does have benchmarks in its repository, the benchmarks I plan on running are in a different repository.  With this goal in mind, I created another Job in Hudson to checkout and run the benchmarks.

Its setup is very similar to that of JRuby, it checks out the source and runs the benchmarks.  The main difference is that a parameter is passed to the project to tell it which Ruby VM to use.  The Parameterized Trigger Plugin is necessary to pass a parameter from one project to another.  The way it works is you set a parameter in the project receiving the parameter near the top of the page.  In my case, I added a RUBY_PATH parameter.  Then you setup the build job to send that parameter to the benchmarks job.

To do this, I went back to the JRuby job and turned on the Trigger parameterized build on other projects option.  It should be the last option down at the bottom of the page.  I set the JRuby job to trigger with the benchmarks job name, and in the predefined parameters field I put the following:

 RUBY_PATH=$WORKSPACE/bin/jruby

After this is in place, when a JRuby build finishes it will start a benchmarks run.  Now that your benchmarks are up and running, the next part to this series will go over how to display the information in a way that makes it easy to spot regressions.

If you have any questions or if I went over something too quickly, post a comment and/or ask a question.