noss in #erlang on freenode recently brought to my attention: Jetty Continuations
This is an interesting blog entry. The basic idea is, instead of using 1 thread per connection, since connections can last awhile, they use 1 thread per request that a connection has. The hope being, a connection will idle most of the time and only send requests once in awhile. The problem that they ran into is, a piece of software is using a request timeout to poll for data. So requests are now sticking around for a long time, so they have all these active threads that they don't want. So to deal with this, they use a concept of continuations so the thread can die but the request still hang around, and then once it's ready to be processed a thread is created again and the request is handled. So having all these requests hanging around that arn't doing anything is no longer a problem.
Well, this begs the question, why are you using a dynamic number of threads in the first place if you are going to have to limit how many you can even make. If the problem, in the first place, is they have too many threads running, then their solution works only for idle threads doesn't it? Being forced to push some of the requests to a continuation means they have applied some artificial limit to the number of threads which can be run. What happens then, when the number of valid active requests exceeds this limit? What then? Push active requests to a continuation and get to then when you have time? Simply don't let the new requests get handled? If they want to to use threads to solve their problem then putting a limit on them seems to make the choice of threads not a good one. Too poorly paraphrase Joe Armstrong, are they also going to put a limit on the number of objects they can use? If threads are integral to solving your problem, then it seems as though you are limiting how well you can solve the problem.
This also got me thinking about other issues involving threading in non-concurrent orientated languages. Using a COL (Concurrent Orientated Language) all the time would be nice (and I hope that is what the future holds for us). But today, I don't think it is always practical. We can't use Erlang or Mozart or Concurrent ML for every problem due to various limiting factors. But on the same token, using threads in a non-COL sometimes makes the solution to a problem a bit easier to work with. At the very least, making use of multiple processors sounds like a decent argument. But writing code in, say, java, as if it was Erlang does not work out. I think the best one can hope to do is a static number of threads. Spawning and destroying threads dynamically in a non-COL can be fairly expensive in the long run and you have to avoid situations where you start up too many threads. I think having a static number of threads i a pool or with each doing a specific task is somewhat the "best of both worlds". You get your concurrency and you, hopefully, avoid situations like Jetty is running into. As far as communication between the threads is concerned, I think message passing is the best one can hope for. The main reason I think one should use message passing in these non-COL's is, it forces all of the synchornization to happen in one localized place. You can, hopefully, avoid deadlocks this way. And if there is an error in your synchornization, you can fix it in one spot and it is fixed everywhere. As opposed to having things synchornized all over the code, god knows where you may have made an error.
I think this post most likely opened up a can of worms. I lightly touched on a lot of issues and most likely did not explain things in full. Perhaps this will raise some interesting questions.
There are many good reasons to control the maximum number of threads a server can use.
ReplyDeleteFirstly QOS concern may mean that you wish to give good service to 1000 users and reject excess users, rather than give crappy service to 2000 users.
For throughput concerns, often it is best not to attempt to handle every thing in parallel. If you are limited (eg by DB connections) then it is often best to attempt 10 serial batches of 100 requests in parallel. This may cost some latency for most requests, but often attempting 1000 requests in parallel may result in more contention and worse throughput and worse latency.
Finally the number of threads a given system can run is finite. If you keep creating threads for offerred load, then you will eventually run out of memory, file descriptors or some other OS resource. It is best to limit your server to not hit these hard limits as this will crash your server. If you don't you are just asking for a DOS attack.
So threads are a key resource that must be managed. Saying they are managed does not mean having a tiny pool of them, but it does mean limiting the maximum number of them and trying to make efficient use of these moderately expensive resources.
If you are out of threads, then new connections and new requests are not initially rejected. They wait in the OS accept queue or TCP/IP buffers. If a thread becomes available before the timeout, then the request is handled. If a thread does not become available then your server is already got more load than it can handle and accepting more load is not going to help anybody!
Hrm, I think you somewhat missed the point of my post. Also sounds like you are unfamiliar with Erlang. But that's ok, I'll try to be clearer here.
ReplyDeleteQoS is very important, obviously. However, I think that the number of threads your machine can run should not be a factor in your QoS. The initial analogy I made was, if you decide that threads is an integral part of how you solve your problem, then you should not have to worry about how many you can make use of when solving your problem.
Since you are using Java for your product, you are obviously making heavy use of objects. What if the Java VM put a limit on the number of objects you can have created at a single time. Obviously you can only have a finite number of objects created (you have finite RAM), but what if the VM imposed a limit well below that of what you will need to solve your problem. Well the same goes for threads, you shouldn't have to worry "Have I created too many threads?".
You call threads "moderately expensive resources". Yes they are, in *non* concurrent orientated languages. This is why I wonder why you are using threads in a non concurrent orienatated language in the first place. In Mozart and Erlang, a thread is about as heavy as a class is in Java. The hard limit may be around 30,000+ threads in a typical Erlang implementation.
So, the basic point of my post is this: Why are you trying to make threads an integeral part of how you solve your problem if your implementation cannot even allow you to make enough threads to properly solve it? Shouldn't you be using soem sort of asynchornous framework, or a concurrent orientated language, where you don't have to be concerned that you are hitting the limit in concurrency?
But objects are a limited resource! When writing a java server application trying to limit the number of objects created is a very important optimization. You minimize your footprint and your garbage collection.
ReplyDeleteSo good java is written with object management in mind just as much as it is with thread management in mind.
That is not to say that other languages do not exist that might free the mind of the programmer more. But my task is to provide a solution in Java - not just provide a solution. I am writing an asynchronous framework (Jetty) so that other programmers can write servlet without concern for threading issues.
To use another asynchronous framework would not be solving the problem, just giving the problem to somebody else to solve.
Thank you Orbitz for writing this article, although it seems not all your your readers corroborate with what you meant by "concurrent oriented languages".
ReplyDeleteI strongly concur that languages *such as* Erlang (I'm saying such as, because Erlang got the concept right, and other languages /platforms/technologies may follow) will lead or at least make the transition into the future easier.
While people are talking about 16-, 32-, 64- bits... And limit their "stuff" (whatever it is, threads, objects, RAM, ...) accordingly, in Erlang there is no such hard limit.
Erlang processes can grow as big as it wants, provided you give it *enough resources*. Which means, the *same* Erlang program can run on 1 node on a single workstation, or on 1,000 servers spread across different buildings (or continents). The programmer doesn't care anyway.
How much limited RAM? How much sockets can be open? etc. doesn't depend on the programmer, and hopefully the programmer won't need to care about it. Who will care about it is the one who'll be deploying and running the Erlang program.
Most people still think of programming (and worse, think of Erlang) as procedural languages, then built things on top of it including threading... a threading framework.
Erlang on the other hand is sort of kernel (hence why it's called a VM, not simply an interpreter but a real VM that manages processes the way a OS manages OS processes). Every function runs on different processes. A process may run in its own Erlang VM node, a different VM node in the server, or on another server. The program doesn't really care that much (it can care, but doesn't have to use a "distributed framework" the way other languages do.)
God I probably should've written my own blog post =))