Fun with processes (Updated)

I really need to go through all of my Erlang documentation so people like petekaz can stop making me feel small.
I had an issue with some poor Erlang code I was running where it performed a file:open but not the appropriate file:close. Not wanting to restart my application I wanted to know how I could fix this problem without shutting it down. Erlang, of course, provides a solution.
One thing that made this workable in the first place was the fact that I had no other open files besides those that needed to be closed.

So imagine for a second you have opened a number of files and lost the reference to them. For example:

(ort_bot@blong)21> [file:open("/tmp/c2.log", read) || X <- lists:seq(1, 10)].
[{ok,<0.8974.7>},
 {ok,<0.8975.7>},
 {ok,<0.8976.7>},
 {ok,<0.8977.7>},
 {ok,<0.8978.7>},
 {ok,<0.8979.7>},
 {ok,<0.8980.7>},
 {ok,<0.8981.7>},
 {ok,<0.8982.7>},
 {ok,<0.8983.7>}]

file:open causes a process to be created (this is Erlang afterall).
In the shell you can simply type i(). and get output like:

(ort_bot@blong)22> i().
Pid                   Initial Call                          Heap     Reds Msgs
Registered            Current Function                     Stack              
<0.0.0>               otp_ring0:start/2                      377     6569    0
init                  init:loop/1                              2              
<0.2.0>               erlang:apply/2                        4181   179350    0
erl_prim_loader       erl_prim_loader:loop/3                   5              
<0.4.0>               gen_event:init_it/6                   1597     1114    0
error_logger          gen_event:loop/4                        10              
<0.5.0>               erlang:apply/2                        6765     8325    0
application_controlle gen_server:loop/6                        7              
<0.7.0>               application_master:init/4              377       45    0
                      application_master:main_loop/2           8              
<0.8.0>               application_master:start_it/4          233       90    0
                      application_master:loop_it/4             5              
<0.9.0>               supervisor:kernel/1                    610     1373    0
kernel_sup            gen_server:loop/6                       12              
<0.10.0>              rpc:init/1                             610 17456475    0
rex                   gen_server:loop/6                       12              
<0.11.0>              global:init/1                          233     1246    0
global_name_server    gen_server:loop/6                       12              
<0.12.0>              erlang:apply/2                         233      275    0
                      global:loop_the_locker/1                 4              
<0.13.0>              erlang:apply/2                         233        4    0
                      global:collect_deletions/2               6              
<0.14.0>              inet_db:init/1                         233      137    0
inet_db               gen_server:loop/6                       12              
<0.16.0>              supervisor:erl_distribution/1          377      307    0
net_sup               gen_server:loop/6                       12              
<0.17.0>              erl_epmd:init/1                        233      137    0
erl_epmd              gen_server:loop/6                       12              
<0.18.0>              auth:init/1                            233       77    0
auth                  gen_server:loop/6                       12              
<0.19.0>              net_kernel:init/1                      233     3272    0
net_kernel            gen_server:loop/6                       12              
<0.20.0>              inet_tcp_dist:accept_loop/2            233      178    0
                      prim_inet:accept0/2                     10              

This list goes on for awhile longer, depending on what you have done. As you scan through this output you should see something like:

<0.8974.7>            erlang:apply/2                         233      135    0
                      file_io_server:server_loop/1             3              
<0.8975.7>            erlang:apply/2                         233      135    0
                      file_io_server:server_loop/1             3              
<0.8976.7>            erlang:apply/2                         233      135    0
                      file_io_server:server_loop/1             3              
<0.8977.7>            erlang:apply/2                         233      135    0
                      file_io_server:server_loop/1             3              
<0.8978.7>            erlang:apply/2                         233      135    0
                      file_io_server:server_loop/1             3              
<0.8979.7>            erlang:apply/2                         233      135    0
                      file_io_server:server_loop/1             3              
<0.8980.7>            erlang:apply/2                         233      135    0
                      file_io_server:server_loop/1             3              
<0.8981.7>            erlang:apply/2                         233      135    0
                      file_io_server:server_loop/1             3              
<0.8982.7>            erlang:apply/2                         233      135    0
                      file_io_server:server_loop/1             3              
<0.8983.7>            erlang:apply/2                         233      135    0
                      file_io_server:server_loop/1             3   

All of those file_io_server's are the open files. And the Pid associated with it is the file handle. So to close all these files all you have to do is call file:close on those pids. You can create a pid give 3 numbers in the shell by using the pid function:

(ort_bot@blong)23> file:close(pid(0, 8974, 7)).                              
ok
(ort_bot@blong)24> file:close(pid(0, 8975, 7)).
ok
(ort_bot@blong)25> file:close(pid(0, 8976, 7)).
ok
(ort_bot@blong)26> file:close(pid(0, 8977, 7)).
ok
(ort_bot@blong)27> file:close(pid(0, 8978, 7)).
ok
(ort_bot@blong)28> file:close(pid(0, 8979, 7)).
ok
(ort_bot@blong)29> file:close(pid(0, 8980, 7)).
ok
(ort_bot@blong)30> file:close(pid(0, 8981, 7)).
ok
(ort_bot@blong)31> file:close(pid(0, 8982, 7)).
ok
(ort_bot@blong)32> file:close(pid(0, 8983, 7)).
ok

Now, call i() again and you'll notice all of those pesky file's have been closed. You can also do all of this from a remote machine if you have run your erl properly. For example my remote application is run on a machine called 'blong' and the node name is 'ort_bot'. I am on 'ooter' on a machine called 'osx'. So from 'osx' I can do this:

osx:~/projects/Erlang orbitz$ erl -sname ooter -setcookie cookiemonster
Erlang (BEAM) emulator version 5.4.13 [source] [threads:0]

Eshell V5.4.13  (abort with ^G)
(ooter@osx)1> 
User switch command
 --> r ort_bot@blong
 --> j
   1  {shell,start,[init]}
   2* {ort_bot@blong,shell,start,[]}
 --> c 2
Eshell V5.4.12  (abort with ^G)
(ort_bot@blong)1> i().
Pid                   Initial Call                          Heap     Reds Msgs
Registered            Current Function                     Stack              
<0.0.0>               otp_ring0:start/2                      377     6579    0
init                  init:loop/1                              2              
<0.2.0>               erlang:apply/2                         610   161416    0
erl_prim_loader       erl_prim_loader:loop/3                   5              
<0.4.0>               gen_event:init_it/6                    610      557    0
error_logger          gen_event:loop/4                        10              
<0.5.0>               erlang:apply/2                        6765     8289    0
application_controlle gen_server:loop/6                        7              

Connecting to a remote shell is easy and we now have a shell on that node and can do anything from here we want (infact I've done halt(). quite a few times by accident not thinking oi!).

Alternativly, you can also use pman, which provides an interface much like i() but does more interesting stuff too. For instance you can see linked processes and kill various processe through pman. In order to use pman, on your local node simply run: pman:start().
This provide you with a window that looks like:



This looks just like i(). And from the 'node' menu you can switch nodes to view the processes on. As you can see from this screenshot I am viewing the processes on ort_bot@blong, which is a remote node.

As you can see, Erlang provides a very powerful set of tools to keep your application stable and perform various operations. It is things like this that make me glad to run code in a shell rather than a stand alone appliction. For situations like this, the power of the erlang shell is unparalleled.

Update:
Well in his infinite wisdom, after a bit of playing around petekaz came up with two functions that makes solving this particular problem all of the easier. My goal was mainy to show how to interact with the shell to get useful information about processes, but petekaz has shown me how to actually close a specific file. It seems file:open drops some useful information into ets.

[ file:close(P) || P <- processes(), {ok, "/tmp/test.erl"} == file:pid2name(P) ].

Where "/tmp/test.erl" is the file you are trying to close.
This makes use of two useful functinos. processes for starters evaluates to a list of all current proceses runninging in the node, and file:pid2name takes a pid and tells you what file it is associated with. If the pid is not a file it returns undefined.
So it seems Erlang has a complete solution to this problem.

Thanks petekaz for figuring this one out completely.

Oort

Well it was pretty painful but I ripped out most of my terrible code and replaced it with pretty gen_server's and the likes. The code is still in a state of flux and needs to be polished a bit more, but it's a good start. You can gawk at it here:

svn://ortdotlove.net/Erlang/trunk/p1

Ignore the manderlbot source tree in there.

MEUG

MEUG is Massachusetts Erlang User Group.
For some reason, there seem to be a lot of Erlang users in MA. I would like to get us all together semi-regularly. We can enjoy some good food, some drinks, and talk about Erlang, or whatever else comes up. Concurrency in general is an interesting topic that that is much to talk about.
I will keep people updated. If there are any MA based Erlangers interested that I am not yet aware of feel free to email me at:
orbitz AT ortdotlove.net (That is really ortdotlove) or post a comment.

Re: Re: Re: Ruby Array class sucks

I have been meaning to respond to this post on the factor blog for awhile. Factor has grown a lot, and this blog tracks the development of it. Factor is the result of Forth and Lisp having babies. But the post I'm interested in doesn't have so much to do with factor but with ruby. It is in response to another post (URL at the factor site) about a complaint with Ruby's Array class. The complaint centers mainly around the Array object having far more methods than it should and quite a few that don't belong there. Slava's (factor creator) response suggests that the methods do deserve to be there and from a functional standpoint most of them make plenty of sense.

For a number of the methods in Array, I do agree with slava that they should be present. For instance, the authors complaint about Array.new being able to preset an array to a specific value is silly to me, I often want some sort of initial value to work from so I would imagine many other people are in my boat.

Around Array.transpose, I start to disagree though. transpose takes an Array and treats it like a matrix. Slava's response is:

It might surprise the author to learn that mathematically, a matrix is a two-dimensional array. In Factor, you can perform mathematical operations on vectors, such as vector addition, dot product, and so on.

Ok, so basicaly the argument is, a matrix resembles an array in some way and factor allows these sort of manipulations, thus ruby should. Well ok, let's say, yes, a matrix is a two-dimensional array, that's nice. But what does that make an Array then? An Array is also a two-dimensional array? That sounds a bit circular. Why should a method in Array depend on Array being thought of as a specific type of object. If we want to do matrix work, I suggest abstracting a matrix type, even if it's as simple as Matrix = Array. But really, it sounds like transpose should be in some sort of Matrix object.

Array.rassoc also seems a bit out of place to me. Slava says:

Yes, and there's nothing wrong with that. Association lists, while they suffer from linear lookup time, are occasionally useful where you want to have a mapping together with a certain intrinsic order. For example, Factor's generic word code takes the hashtable of methods defined on a generic, converts this to an association list, then sorts the association list according to the partial ordering of classes. Finally, the sorted association list is used to generate code.

Again, the argument of "This is ok because factor does it" does not quite fly with me. Now, when I write code and I'm not quite sure what data type I really want, but I need some sort of key->value association, I will often just use a list or array type to start out with, abstracted into some interface so when I decide what I finally need I can simply change the backend. Now, is this a good reason to put a function that treats a list like a map inside a list object? In my opinion, No.

In my opinion, I would use some sort of generic sequence object that has various methods that act on arrays and lists and other sequence objects, allowing you to perform some of these manipulations. In most languages, I would probably go as far to even make something like rassoc just be a plain function, no need to make it part of an object. But I have a feeling a lot of Ruby programmers don't feel content unless they have methods in some class.

This Array object seems to be similar to various classes in Python. Back in the day, Python developers could add methods to classes without any sort of vote. This is why the Python standard lib has lots of random modules that one would not normally think should be in a standard library, and also why Python has been making such an effort to clean up a lot of the standard classes and provide a more consistent interface to them.

Sorry this post does not actually contain anything on erlang. My next one will, I promise. Feel free to flame me in response to my Ruby thoughts, just be civil.

Port Drivers

Does anyone have any good tutorials on writing ports? Google doesn't seem to return much on initial tries.

Single exit point

Many programmers insist on only having the flow of control in their function end at a single point. I got wondering for a few minutes if any languages enforce this style of programming. Then I realized. Erlang has no 'return' operator, so doesn't Erlang enforce this? That sounds about right as far as I can see. So an Erlang function can only have 1 exit point. That should make them feel good.


But as was just pointed out to me in the middle of writing this, Erlang has 'throw'. There can be many of thoughs. So perhaps Erlang doesn't have 1 exit point.

On a final note:
Does having 1 exit point even matter? Who cares? The beauty of Erlang (in my opinion) is it almost forces you to write short concise functions. Every single one of my erlang functions fits on 1 screen easily, so does having multiple exit locations from there matter? In my opinion, no. I don't need to go searching through pages of code for that function to find it, I can see it, so who cares?

The Twisted Matrix

It has been a long road from where I started to using Erlang. I certainly don't plan on stopping with Erlang but so far I am quite the fan of it. On this road, which finally lead me here, I have used C, C++, Java, and Python, to name a few. I still have code for a multithreaded http I was writing in C hanging around.
I still use Python quite a bit, and one of the best tools in Python I have used is the Twisted Matrix framework. Prior to discovering Erlang (well more of introduced to it by noss, which i am deeply in debt for), I felt that Python was really *the* language for socket programming. Twisted is what made me think this.
Now, just to be clear, Erlang certainly does not make Python obsolete. Don't confuse enthusiasm for Erlang with feeling every other language is useless. I introduced Python and Twisted into our development cycle at work and I am fairly pleased with the results.
For those that don't know what Twisted is, it is a framework for developing software. What makes Twisted special is they have put a lot of work into making everything work in one thread. Making a server or client that handles multiple network connections is fairly trivial in Twisted.
So what makes Twisted work how it does? A uniform eventloop. In their case it's called a reactor. There are a bunch of different kinds of reactors in Twisted. One needs to be made for whichever kind of event loop you are using. There is a select reactor (default), gtkreactor, gtk2reactor, wxwidgets reactor, and some sort of win32 reactor (I think still under development). The good thing is, most of your code can generally be written without caring about what reactor is being used. That is, atleast, the portions that don't depend on gtk or win32 or wx.
Five years ago, when the Twisted project was first created, the authors feel that Python was the best language for the job. My question is, if Twisted was started today, would Python still be the best choice. I address part of the problem in a previous post. Part of the problem that I point out is, you have to do a lot of work to force everything into the event loop. That makes code reuse more difficult, and near impossible if you need to use a proprietary library that you can't make non-blocking.
An obvious example of this is the adbapi module. adbapi module can use any DB-API 2.0 compliant module. This is to make adbapi useful. There are quite a few DB-API 2.0 compliant modules for Python so the best thing is to make use of them. adbapi's API is not asynchronous or non-blocking though. Doing queries in the Twisted reactor thread means the entire application will stall until the query is finished. To solve this, Twisted does the queries in a thread. Twisted has to fall-back on the very thing it is trying to avoid in order to work. Python is particularly bad at threading too so Twisted tries to keep the number of threads to a reasonable amount. This means the number of queries you can have going concurrently is dependent on this.
A native implementation of the postgresql protocol has been implemented. It is called pgasync. I beleive the author has commented on how much quicker a large number of queries runs in his implementation, although I cannot find the quote. But has the same problem I pointed out before. Because Twisted is all in one thread, everything has to conform to its event loop making the implementation of the pgsql protocol useless. It had to be rewritten to work well in Twisted. This implementation is also fairly useless anywhere except Twisted. I cannot use this in an application that uses an event loop other than Twisted. Wouldn't it be nice if I could use code between projects that arn't dependent on Twisted? In a COL this is a non-issue. Simply run the database code in a process. Processes are part of the language so they are part of any program.
So, to get back to my question, would Python still be the choice of Twisted today, I don't know. I asked the people of Twisted if they still would. For the most part, it seems like they didn't quite know Erlang well enough to say if they would choose it. A few developers said one of the main reasons they chose Python was for its large standard library. In particular the 'os' module was pointed out as an important module that makes Python a good choice. However, after some research, most of the 'os' module is in Erlang's standard library as well. In my opinion, I think the quality of the Erlang standard library and OTP is quite a bit better than Pythons. Python's standard library seems to have suffered from a time period where anyone put anything they wanted into it. As a result, it is large, but a number of modules are fairly poor in quality.
Even if Python has a number of modules in its standard library that are nice for building Twisted, they still need to do a lot of work to force all of this code into the Twisted event loop. If you choose a COL, then you don't have to do any work in forcing code into a single event loop. Instead, you have to do work to build the standard library that Python has. Is that a lot of work? Yes.
One could also argue, "There are a lot more Python projects than Erlang ones so there is a lot of code being written for Python that Twisted can make use of". I think there is little contest in the point that there are more Python projects and programmers. But can Twisted really make use of this code? I think, in general, no, not without work. A lot of interesting things are probably going to involve some blocking somewhere which either needs to be pushed to a thread or rewritten not to block, such as the example I have already given.
The choice to use Python is a difficult one, I think. In my opinion, Python really does not offer much that Erlang doesn't. I think, considering the services Twisted is trying to offer, a COL would be an excellent choice for it. Twisted does a lot of work trying to make asynchronous programming less confusing but writing concurrent programs is more natural. It feels more natural atleast.
I have had a lot of success using Twisted. I don't think I'd write a socket program in Python without it. I also think I would generally not write a socket program in Python these days. Twisted and Erlang seem to be trying to accomplish much the same thing, although going about things in very different ways. In the end I think something like Erlang will win out. A COL seems to be saying, the world is concurrent so lets try to let people write in this natural way. Whereas, Twisted is forcing people to write in a rather artifical and unnatural way. Asynchronous programming works once you grasp it but even still the flow of the programs is rather awkward and hard to grasp I think.
I feel that, today, something like Twisted would be better off written in a COL. Let the work go into making powerful tools instead of massaging prewritten code into the event loop.