Waiting For Torquebox / JBoss to Start

- jgilman - InfrastructureRails

We recently started to use Torquebox on one of our projects. Torquebox is a Ruby application platform built on top of the JBoss Java application server. It has a lot of great features that you can read about on their website torquebox.org. Switching from a custom coded Tomcat environment for deploying JRuby applications to Torquebox has had it’s share of challenges. We have to deploy a mix of pure Java and JRuby applications. Oddly enough, it was harder to port the Java applications to run on JBoss then it was to run Ruby applications on Torquebox.

One of the challenges I encountered was detecting when JBoss had completely started up and all of our deployed applications were ready. We have a project built on top of Capistrano that orchestrates the deployment of all of our applications across multiple nodes. It has to wait until the applications are completely started on a node before it can verify they are running correctly. JBoss starts very quickly and will respond to HTTP requests while deployed applications start in the background. If an application isn’t fully deployed your request will return an error. I wanted a way to tell if the application server was started that didn’t rely on knowing which applications were deployed in a particular JBoss instance.

JBoss Command Line Interface (CLI)

I found a way to do this using the JBoss Command Line Interface (CLI). The CLI is an interactive command line tool that allows management of a JBoss server (or cluster of servers) from the command line. It’s started from a shell by running bin/jboss-cli.sh. Once started it’s a lot like entering a subshell. It has commands for exploring subsystems similar to a bash shell like ls and cd. It can be used for many different tasks like configuring new datasources, deploying applications, or reading statistics from a datasource. Here’s what reading statistics from a datasource looks like

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[standalone@localhost:9999 /] /subsystem=datasources/data-source=ExampleDS/statistics=pool:read-resource(include-runtime=true)
{
  "outcome" => "success",
  "result" => {
    "ActiveCount" => "2",
    "AvailableCount" => "20",
    "AverageBlockingTime" => "0",
    "AverageCreationTime" => "62",
    "CreatedCount" => "2",
    "DestroyedCount" => "0",
    "MaxCreationTime" => "64",
    "MaxUsedCount" => "2",
    "MaxWaitTime" => "0",
    "TimedOut" => "0",
    "TotalBlockingTime" => "0",
    "TotalCreationTime" => "124"
  }
}

I can list the currently deployed and started applications in JBoss by running ls /deployment at the CLI prompt. I did some experimentation and found that during startup JBoss doesn’t list any applications until they’re all started. My approach is to repeatedly list the deployed applications on the server until it returns at least one application. Once any applications are listed then I know that the server and all the deployed applications are ready for use.

Invoking from Capistrano

I needed to script this from Capistrano. CLI has a non-interactive mode where commands can be passed in on the command line. This is a rough approximation of the Capistrano code that we run.

1
2
3
4
5
6
cli_bin = "#{torquebox_home}/jboss/bin/jboss-cli.sh"
response = ""
run "#{cli_bin} --connect --command='ls /deployment'" do |channel, stream, data|
  response += data
end
started = response.strip.size > 0

There are a lot of different ways to interact with JBoss to accomplish the same thing. There’s even a Java API that you can script from other JVM languages that look really interesting. This method works well enough our purposes and integrates fairly well with our existing Capistrano deployment code.