Interesting Java Jersey + RxJava finding

Posted by & filed under , .

network pc monitorsIt so happens that in a few of the apps I work on here in Netflix we use Jersey libraries inside a Tomcat container. As such I use the JSR annotations a lot safe in the knowledge that Jersey will take care of all the plumbing work and deal with routing and serialization/deserialization so I can concentrate on just writing the actual business logic in my web application.

Apart from the (very common) @Path and @GET @POST @Produces and @Consumes I find myself occasionally to need access to the actual raw HttpServletRequest object — typically this is in order to parse some headers and/or read cookies but I guess for the purpose of this exercise you can imagine any other operation you might need to apply to the request object. Point being is that in such situation I use the @Context annotation and have Jersey automatically inject the object I need (be it the request, response on the SecurityContext etc.) And this is where I came across this obscure yet super-interesting issue. It’s not a bug by the way, looking in fact in the Jersey doccos it seems this behaviour is actually documented. However, it seems its rather little known.

Let’s look at this (very simple) code — as I said run in the context of Jersey + JSR-311 inside a Tomcat container. (Those details don’t really matter — apart from the Jersey bit — however it might make it easier for you to set up fully a project to repro this if you need to; also, it might make it easier to identify if you are hitting exactly this problem when you are reading this.)

import javax.inject.Singleton;
import javax.servlet.http.HttpServletRequest;
import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
 
@Singleton
@Path("unsecure")
public final class UnsecureResource {
   @Path("language")
   @GET
   @Produces({MediaType.TEXT_PLAIN})
   public String usingRequest(@Context HttpServletRequest request) {
      return request.getHeader("accept-language");
   }
}

This class is a simple REST resource which does something very simple: when the user hits the /unsecure/language HTTP endpoint it simply prints out the value of the user’s browser Accept-Language header — in my case this is something like this: en-US,en;q=0.9

Nothing spectacular, right? The request object gets injected by Jersey, I’m calling a method on it and displaying the result.

Now let’s throw RxJava into this. To do this I’m going to basically take the request object, emit it via Observable.just() and then go from there:

@Path("language-rx")
@GET
@Produces({MediaType.TEXT_PLAIN})
public String usingRxRequest(@Context HttpServletRequest request) {
   return Observable.just(request)
      .map(req -> req.getHeader("accept-language"))
      .toBlocking()
      .single();
}

So now if I hit (GET) /unsecure/language-rx I get.. well the same thing! So far not surprised, right?

OK well the surprise come if you start running this in a multi-user environment — if you get multiple users hitting that endpoint at the same time you will have a surprise… but I don’t want to ruin it for you:) Instead I will tell you exactly how to reproduce it for yourself, without the need for other users.

In this case the only operation I am applying on the request object is getHeader — but let’s say that I know I will be doing all sorts of things to the poor object 😀 In that case, in an Rx environment I would probably indicate to the framework that I’m hammering this and I probably need to run this in the computation scheduler (though you can try this with the I/O scheduler or another one). So the above code will just become:

@Path("language-rx")
@GET
@Produces({MediaType.TEXT_PLAIN})
public String usingRxRequest(@Context HttpServletRequest request) {
   return Observable.just(request)
      .map(req -> req.getHeader("accept-language"))
      .subscribeOn(Schedulers.computation())
      .toBlocking()
      .single();
}

Now let’s hit our endpoint again … and:

HTTP Status 500 – java.lang.IllegalStateException: No thread local value in scope for proxy of class com.sun.proxy.$Proxy101

type Exception report
message java.lang.IllegalStateException: No thread local value in scope for proxy of class com.sun.proxy.$Proxy101
description The server encountered an internal error that prevented it from fulfilling this request.

Wooow!!! What gives?

Well it turns out that as per our request, our Rx code is now running in a different thread and when it is trying to access our HttpServletRequest object we get that nasty error.

Looking into this it turns out that in the case of the @Context annotation Jersey will inject a proxy object which uses a ThreadLocal (see here) so when your proxy is finally evaluated it tries to access the ThreadLocal instance — but in this last case, we are by now in a different thread (the Rx computation thread), and in this thread our ThreadLocal instance has NOT been intiatialized (because the Jersey injector does not run in this thread), so we end up with this error.

So how do we get around this? Well there are 2 ways: first one, as shown above, is to only use the HttpServletRequest object in the same thread as the request came in. In this case, simply extracting this operation outside of Rx might suffice as shown in the first version of this.

However, that is not always viable: in the above case we only call one method on the request object so it’s easy to extract this, but what if we had tons of lengthy processing applied to it? In that case definitely we want that wrapped in an Observable.defer() and run in the correct scheduler (computation or I/O or whatever is suitable). And if we want to do that then the trick is to have Jersey inject in fact the ThreadLocal directly and materialize it upfront:

@Path("language-rx")
@GET
@Produces({MediaType.TEXT_PLAIN})
public String usingRxRequest(@Context ThreadLocal<HttpServletRequest> request) {
   final HttpServletRequest r = request.get();
   return Observable.just(r)
      .map(req -> req.getHeader("accept-language"))
      .subscribeOn(Schedulers.computation())
      .toBlocking()
      .single();
}

In this case, we’re ensuring we are getting a handle of the request object in the same thread as the request by extracting it outside all of the Rx operations — this is a very fast operation so it’s not that costly — and then throw that into the Rx chain, as from there on we are dealing with a “materialized” instance of a raw request.

This goes even deeper: I have actually found this because of my usage of RxJava but at the core the issue occurs because of the evaluation of the HttpServletRequest (proxy) in a different thread. So having your own thread pool and passing this proxy into one of your threads in there would render the same issue. In fact, depending how you implement it, it might NOT throw any exceptions but you could end up with the very first request object being used to initialize the ThreadLocal and then consecutive calls. Needless to say this would be even more difficult to track down and detect as you would get some weird errors in some other parts of the code and would take you ages before you realize it. There are pros and cons to this approach and I don’t know if Jersey will ever change this part of their implementation, but for now just be weary of this if you use any other thread apart from the original one for processing requests.

More from my site