Using OS-level Signals in Java for Monitoring

Posted by & filed under , .

DukeTubbingSmallFor those of you familiar to monitoring Java applications, I’m sure the first thing that springs to mind is JMX. And I agree, that is a nice framework to provide various app insights for tracking and monitoring. The trouble with JMX though, if you haven’t got a system in place already to collect this data (like Zabbix or Nagios or the likes), it can be a tedious process to get that up and running only to collect occasionally some simple stats from your application.

Of course, you can go via the route of writing some Java or Groovy code to connect to your app and query it via JMX but you still have to spend some extra time (having written the app) to write the JMX client to query it. Which, I personally find at time annoying — I’d much rather dump the data into a log file and keep tail-ing and grep-ing that file for checking the data. For anyone who has been involved in devops I’m sure you’ll agree that’s a much easier and convenient approach. The trouble with doing that is if you regularly pump out the data into a log files, these things keep growing — and then you have to regularly clean them up, rotate, etc.

Instead, I would prefer a solution where at regular intervals I can query the data (preferably no log writing needed) and act on it — e.g. spawn another process or kill a process and so on. As I said, this can be done with JMX but I find it cumbersome — instead, would be good if the process itself only “logs” data when requested — and in doing so uses the stdout so I don’t have to worry much about cleaning up logs.

So I started looking at what would be an easy way to dump to console when needed a JMX bean. It turns out that the Sun Oracle JDK actually offers support for OS-level signals via their sun.misc.Signal class. So the idea came right away, what if I write a little signal handler which simply dumps out a managed bean to console (using toString() )? This way, a Java app can still offer data / stats via JMX as it would normally, and as such a proper monitoring system can be used where available; while at the same time, you can simply send a signal to the process and get the data in text format on console, then run a quick parser (grep / awk / sed whatever) if needed to get the data.

To try this out, I’ve created a small Java project — available on GitHub here: https://github.com/liviutudor/JavaSignal.

The idea here is simple: I’ve created a basic managed bean (MagicBean) which has just 2 properties:

  • Uptime — gives the time in milliseconds since the instance was created
  • Requests — keeps track of the number of requests issued to this bean. This is a simple counter which gets incremented when the method request() gets called on the bean. (Imagine this method performing all sorts of other “stuff” and at the end incrementing this counter. In this example, the method just increments the counter and exits though.)

The bean provides all the necessary code for a dynamic MBean so it can be easily exported via JMX. On top of that though, I’ve implemented a simple toString() which basically shows a concatenation of Requests and Uptime property — so this can be easily dumped to console and parsed easily if needed.

From there on, it turns out to be rather simple achieving what I was talking about — and this article helped too. The propriety sun.misc packages offer a simple way to deal with OS level signals: simply register a handler via Signal.handle() with an OS-level signal, which needs to be specified by name. Some of these signals you will find out are not available as there is code in the JVM which uses them, so you have to figure out which ones your JDK allows to use. I’ve found out that in most cases USR2 is available so I’m using this in my code. However, see this code here too, where the author uses USR1 as well.

As such,I’ve simply provided a PrintOutObjectHandler — which is just a signal handler that takes an object and when receives the signal prints out the object to the console. In our case, the object is the MagicBean instance. Registered this as a signal handler for USR2 and if you start the process you can “ping” it via USR2 signal:

kill -s USR2 PID

And you will see on console something like this:

Uptime: 1005ms, Requests: 1

(You probably noticed that to make this more “realistic”, in the main app, I’ve got an infinite loop which keeps calling the request() method of our bean, to increment the number of requests — such that the data changes in between calls and we can see that this method of querying data is working.)

This, for me, provides a cheap yet effective way to query apps for data occasionally — when JMX proves to be too expensive to implement. However, since I’m getting the hang of using OS-level signals now, I didn’t stop here!

I had often encountered the situation where I have to run one of my apps as a daemon — in the background. In these cases, you have to sort out the mechanism to start and stop the application. While starting is not a big deal, since a simple shell script providing classpath, parameters etc covers it, stopping the app can be tricky. If your app can be stopped at any point via a KILL signal then that’s fine, nothing to worry about, but if you need a clean way to do this (close files, commit/rollback transactions etc) then this method is not good enough. At which point you need a way to request your app that it shuts down.

Typical approaches are to have a thread listening for a connection on some port, and when a command is received simply shut it down. However, that has a few disadvantages, one of them being the fact you have to write the “server” code to listen on the port, as well as the client code which connects to it and sends the commands to shut down the app.

Of course, you can use something like jsvc for this but I find this needs an extra bit of API added to the app. So instead, since I’m already using the OS signals, I thought why not have a signal sent to the app, capture it and react to it by cleanly shutting down the application?

So I’ve put together an ExitHandler class to take care of that. The idea is simple, we offer an ExitSignalListener interface — all classes which need to shut down cleanly implement this and take actions on notifyExit(). This can be closing database connections, flushing client connections and so on. Then when the app receives the signal (I’ve chosen ALRM in this case — but again, you can use any other) it calls the ExitHandler instance which then notifies all the registered handlers and once this is done it exits the app.

Altogether, with one single mechanism (OS-level signals) and no extra code (since we use classes provided in the JDK) we get to cater for 2 separate needs which otherwise require extra infrastructure or frameworks. I find this pretty neat for small apps — though I will accept the argument of not being suitable for larger applications, where you want more rigorous monitoring. Also, bear in mind that sun.misc is an Oracle JDK proprietary package — it won’t work with OpenJDK for instance; it will also spawn a whole bunch of warning during compilation. (However, it also seems this package was kept in JDK 6 as well as 7 and 8 so chances are it will be there for a while!)

Still, as I said, small feature (and small code!) but with big results — I like it!

The code for this article, in case you missed it, on Github, here: https://github.com/liviutudor/JavaSignal