Joe Freeman's Weblog » Erlang http://joefreeman.co.uk/blog Musings on Software Development, etc Sat, 27 Mar 2010 15:16:35 +0000 http://wordpress.org/?v=2.8.1 en hourly 1 Handling Multipart Uploads with MochiWeb http://joefreeman.co.uk/blog/2009/12/handling-multipart-uploads-with-mochiweb/ http://joefreeman.co.uk/blog/2009/12/handling-multipart-uploads-with-mochiweb/#comments Sat, 12 Dec 2009 00:02:40 +0000 Joe http://joefreeman.co.uk/blog/?p=262 I was getting along fairly well with my own custom-built, lightweight Erlang web server, until I needed to handle file uploading. At this point, I decided it was time to stop trying to re-invent the wheel (or take the lazy approach—whichever way you choose to look at it), and take another look at MochiWeb, which implements this feature.

However, MochiWeb’s documentation is non-existent (unless I’m missing something—please let me know if I am). I came across James Gardner’s post on handling file uploads, which helped me out. But after digging through the MochiWeb source-code I discovered a slightly easier way of handling file uploads. It may be that this method has been added since James’ post in 2007, or maybe it’s just less appropriate to James’ requirements.

I’m going to quickly run over how to setup MochiWeb. I know this has been covered quite a lot by other people, but it’s something a found a little daunting when I started out, so I want to emphasise that it’s a lot easier to setup than maybe first appears. Then I’ll post and explain a short chunk of code for handling uploads, and saving them to /tmp. Finally, I’ll provide an example of using this technique to provide a very simple photo gallery system.

Setting up MochiWeb

There’s a pretty good tutorial on the BeeBole blog, but I’ll cover the basics here.

First step is to check out the code from the Google Code repository. Make sure you have subversion installed, and then:

svn checkout http://mochiweb.googlecode.com/svn/trunk/ mochiweb

Now, make sure that you have Erlang installed, and:

cd mochiweb
make
chmod +x scripts/new_mochiweb.erl
./scripts/new_mochiweb.erl mochiweb_uploads ../.

This will build the MochiWeb system and then create a new MochiWeb project. You can repeat this process whenever you’re starting a new project. Note that mochiweb_uploads is the project’s name. MochiWeb is a bit picky about project names: they must be valid module names, so (it seems) you can’t use hyphens.

Next, we should make sure that the project builds and runs before we start coding:

cd ../mochiweb_uploads
make
./start-dev.sh

By default, your MochiWeb project is setup as a lightweight web server that serves files from the priv/www directory on port 8000. All being well, you should now be able to point your browser at http://localhost:8000/ and see the ‘MochiWeb is running.’ message.

If you have any problems, you’ll need to wade through the progress reports on the console to try and figure out where things are going wrong. The clues will be in any ‘crash reports’.

You can stop the server with Ctrl+C, then entering ‘a’, and hitting return. Or you can just hit Ctrl+C twice.

Handling File Uploads

First of all, we need to put together a page that we can test our uploading with. We’ll just replace the index.html file in priv/www with something straight-forward:

<html>
<head>
<title>MochiWeb Upload Test</title>
</head>
<body>
<form action="upload_photo" method="post" enctype="multipart/form-data">
    <input type="file" name="photo" />
    <input type="submit" value="Upload" />
</form>
</body>
</html>

Routing the Request

The code for starting the web server is in src/mochiweb_uploads_web.erl. Open up this file and take a look at the loop/2 function. The function is split up into two parts by the outer case construct: the first part is for HTTP GET (and HEAD) requests, the second is for POST requests. We’re going to be handling the file uploads in the POST clause. Like so:

'POST' ->
    case Path of
        "upload_photo" ->
            upload_photo(Req);
        _ ->
            Req:not_found()
    end;

All we are doing here, is delegating responsibility for handling the ‘upload_photo’ request to a function called ‘upload_photo’, which we can add to the bottom of mochiweb_uploads_web.erl:

upload_photo(Req) ->
    Req:ok({"text/html", [], "<p>Hello, world!</p>"}).

That’s going to just return a ‘Hello, world!’ message so we can check the request is getting routed correctly. If we try out the system (make, ./start-dev.sh, go to localhost:8000), we should initially see the form, then after submitting the form our message should appear.

Handling the POSTed multipart data

Now onto actually handling the POSTed multipart data. We’ll replace our upload_photo/1 function with this:

upload_photo(Req) ->
    FileHandler = fun(Filename, ContentType) -> handle_file(Filename, ContentType) end,
    Files = mochiweb_multipart:parse_form(Req, FileHandler),
    Photo = proplists:get_value("photo", Files),
    % TODO: handle the photo here
    Req:ok({"text/html", [], "<p>Thank you. <a href=\"index.html\">Upload another?</a></p>"}).

Let’s examine this line-by-line.

First of all, we’re specifying a function that we pass to the mochiweb_multipart:parse_form/2 function. The function that we’re passing will get called once for every file that is present in the POST data (i.e., that is in the form)—don’t forget that multiple files may be being uploaded.

The file will be split into ‘chunks’. So it’s the job of this ‘FileHandler’ function to return another function that will be used to consume each chunk of the file (and then, finally, the ‘eof’ atom). We’ll come back to this in a moment.

The parse_form/2 function will then return a list (in fact, a property list) of all the files. There will be a mapping from the name of the input to the value finally returned from our file handler.

Let’s take a look at the handle_file/2 function:

handle_file(Filename, ContentType) ->
    TempFilename = "/tmp/" ++ atom_to_list(?MODULE) ++ integer_to_list(erlang:phash2(make_ref())),
    {ok, File} = file:open(TempFilename, [raw, write]),
    chunk_handler(Filename, ContentType, TempFilename, File).

We use erlang:phash2/2 and make_ref/0 together with a (hopefully) application-specific prefix to construct a random filename which will reside in ‘/tmp’. A couple of points to make here: collisions aren’t impossible and this is platform-dependent. We could get the temporary directory from the operating system, and we could generate better filenames.

We open a file for writing to and pass it to our ‘chunk handler’ ready for consuming the first chunk. Note that chunk_handler/4 is a function that returns a function. We return this generated function back to the MochiWeb code, which will use it to handle the first chunk.

Before I go on to explain the ‘chunk_handler’, I should point out that in theory we could use this opportunity to filter the input. If (for some reason) we wanted to handle different types of file in different ways, we could do this based on the ContentType. Annoyingly, there’s no way to inspect the name of the field (i.e., the name of the HTML input) that we’re handling at this stage.

As I mentioned, the chunk_handler/4 function simply returns another function (which will accept one parameter) that MochiWeb will call once it has fetched a chunk. I’m using the term ‘chunk’ casually: it can be either a raw chunk of data from the socket, or the eof atom. If the chunk being passed is not the eof atom, the function must return another function which will be used to handle the next chunk. However, if the function is passed the eof atom, we will return a value that we want to associate with this part of the multipart data. The value returned here will be the value that is stored in the property list we mentioned earlier (in upload_photo/1).

Here’s our chunk_handler/4:

chunk_handler(Filename, ContentType, TempFilename, File) ->
    fun(Next) ->
        case Next of
            eof ->
                file:close(File),
                {Filename, ContentType, TempFilename};
            Data ->
                file:write(File, Data),
                chunk_handler(Filename, ContentType, TempFilename, File)
        end
    end.

So, don’t forget we’re returning a function here, which will get executed later on. There are two cases, as we’ve discussed:

  • The eof case involves closing the file, and returning a tuple containing the original filename, the content-type and the temporary filename. We could return this in record form, or we may choose just to return the temporary filename if we didn’t care about the other parameters.
  • The non-eof case involves writing the chunk of data to the file, and then returning a function to handle the next chunk of data.

This concludes the data-handling code. At this point you may wish to refer back to the upload_photo/1—this is where you would then be able to reference each of the uploaded files in the property list by it’s HTML input’s name. The value in the property list would be the {Filename, ContentType, TempFilename} tuple being returned by chunk_handler/4.

A Semi-practical Example

To turn this into something ‘useful’, I’ve put together a rudimentary photo gallery system.

We will adapt our upload_photo/1 function to take another two parameters (the directory for the photos and a list of valid file extensions; the function will hence become upload_photo/3).

The total of our security features will be to check that the extension of the file corresponds to a recognised image type. Anyone will be able to upload files.

Here’s our function:

upload_photo(Req, PhotoDir, ValidExtensions) ->
    FileHandler = fun(Filename, ContentType) -> handle_file(Filename, ContentType) end,
    Files = mochiweb_multipart:parse_form(Req, FileHandler),
    {OriginalFilename, _, TempFilename} = proplists:get_value("photo", Files),
    case lists:member(filename:extension(OriginalFilename), ValidExtensions) of
        true ->
            Destination = PhotoDir ++ OriginalFilename,
            case file:rename(TempFilename, Destination) of
                ok ->
                    Url = "/",
                    Req:respond({302, [{"Location", Url}], "Redirecting to " ++ Url});
                {error, Reason} ->
                    file:delete(TempFilename),
                    html_response(Req, "An error occured whilst trying to move your file: " ++ atom_to_list(Reason) ++ ". Does the destination directory exist?")
            end;
        false ->
            file:delete(TempFilename),
            html_response(Req, "Invalid file type. File extension must be one of: " ++ string:join(ValidExtensions, ", ") ++ ". <a href=\"/\">Try again?</a>")
    end.

You can download the rest of the code. It’s not very elegant, but I was keen to put together something for you to take home.

Obviously this isn’t the sort of thing you should consider making available to the world wide web, but hopefully it’ll be useful in explaining how to handle file uploads.

]]> http://joefreeman.co.uk/blog/2009/12/handling-multipart-uploads-with-mochiweb/feed/ 1 Deploying an Erlang Appliance with SUSE Studio http://joefreeman.co.uk/blog/2009/08/deploying-an-erlang-appliance-with-suse-studio/ http://joefreeman.co.uk/blog/2009/08/deploying-an-erlang-appliance-with-suse-studio/#comments Tue, 04 Aug 2009 08:27:54 +0000 Joe http://joefreeman.co.uk/blog/?p=123 Having read Nat Friedman’s blog post about the release of SUSE Studio, I thought I’d take a look at it. I was also interested to find out how easy it is to create an appliance for a simple Erlang application.

suse-studio

An Introduction to SUSE Studio

I’m very impressed with SUSE Studio. It allows you to completely customise a Linux distribution and actually try it out (in the browser!) before you download it. This is, in my opinion, a pretty novel concept. At least at this scale.

Being able to have this control before you download the distribution obviously saves you from downloading a load of components that you don’t want. But the benefits don’t stop there:

  • Performing the administrative tasks of setting up the system can be performed before installation. This means a system can be deployed many times with minimum effort. Products like Xen and VMware provide cost-effective ways to host several different systems together and keep them isolated from each other. If one of your applications breaks, the others are (potentially) isolated from the problem.
  • Applications can be deployed and run in a pre-specified environment. This means their behaviour will be more predictable, and it will be easier to provide support for them. If a client is reporting problems, it will be much easier to replicate them yourself in a test environment.

My Experiences with SUSE Studio

The service is currently by invite only. But having filled out the form, I got an invite the next day:

Thank you for your interest in SUSE Studio, a new service that makes it quick and easy to create and test software appliances from your web browser. We are pleased to invite you to try SUSE Studio.

Sign in is via OpenID, which is nice.

The interface is very smooth and pleasant to use. Helpful messages appear in the left-hand column, which let you know where you’ve gone wrong, or give you tips as you go, and there’s a helpful ‘undo’ feature. The site is also scattered with a selection of pleasing cartoons.

The ‘test drive’ feature is awesome. Once you’ve finished configuring the system you need to build it, which takes a little time. When this is completed, just click on ‘Testdrive’, and a flash applet lets you watch the machine and interact with it (via VNC). Nat has an article about some of the implementation details.

By enabling networking in test drive, you can also connect to the machine via SSH, or with your web browser if there’s a web server running.

Whilst Test Drive is running, you can even see a list of the files that have been modified, view diffs and download the modified files.

A impressive number of configuration parameters are at your disposal. You can even personalise the system by changing the logo and background image.

I noted a minor problem with the interface so I sent some feedback to the developers to check they were aware of it. I was impressed that they replied to me about ten minutes later (to say they were aware of the problem, thanks).

Setting up an Appliance with Erlang

The steps involved in setting up a basic appliance are very simple. I’ll skim over these steps and instead focus on those involved in setting up Erlang, and adding your application to the appliance. The aim is to have an Erlang application boot automatically when the appliance is started.

I started off with the openSUSE 11.1 ‘just enough’ base template, which, for reference, has the following packages installed:

aaa_base, branding-openSUSE, dhcpcd, grub, hwinfo, insserv, kbd, kernel-default, licenses, mkinitrd, module-init-tools, netcfg, openssh, openSUSE-build-key, openSUSE-release, PolicyKit, polkit-default-privs, procps, pwdutils, rpcbind, rpm, sysconfig, syslog-ng, vim

The next step is to add a repository that contains the Erlang package. Select ‘Add repositories…’ from the Software page, search for ‘erlang’ and then add the suggested ‘devel:languages:erlang 11.1′ repository. At time of writing, this will install Erlang release R12B-5.

Then, from the software page, search for ‘erlang’, and select the ‘erlang’ package to be added.

Then make a few more configuration changes as required—for example, selecting the keyboard layout and time zone, and adding users (with default passwords that should obviously be changed following installation).

The next step is to add the Erlang application. I won’t go into details of setting up OTP applications, but I will explain how to automatically generate a bootable release file.

The release file needs to be setup to work with the installed version of Erlang, and the boot script needs to be generated in the same system environment (so we know we’ve got the correct versions of applications). To achieve this, I have written an Erlang script to be run when the appliance is first booted, which will generate a release file and accompanying boot script.

The main function that generates the release file is:

generate_release(Name, Version, Apps) ->
    ErtsVsn = erlang:system_info(version),
    lists:foreach(fun(App) -> application:load(App) end, Apps),
    AppsVsns = lists:map(fun(App) -> {App, get_app_vsn(App)} end, Apps),
    {release, {Name, Version}, {erts, ErtsVsn}, AppsVsns}.

Download the complete script to try it out.

The application I am deploying is a simple echo server, but I won’t go into details here.

You need to package up your application (possibly including the compiled make_release.erl script), then upload it under the ‘Overlay files’ section, and specify a directory to extract the files to. I put the application into /usr/local/, and use the following directory structure:

directory-structure

(Note that the echo-1.0.* files wouldn’t be there when uploaded as they get generated by the script.)

If you’d like the application to be started when the system boots, you may wish to add this to a boot script. Boot scripts can be configured under the ‘Configuration’ section. The following boot script will generate the release file when the system is booted for the first time. Following this, and on subsequent boots, the the Erlang application will be started.

My boot script looks like this:

if [ -f /etc/init.d/suse_studio_firstboot ]
then
  # Generate release for Echo Server
  echo "Generating release for Echo Server..."
  erl -noshell -s make_release -pa /usr/local/Echo-1.0/ebin/
fi
 
# Start the echo server
echo "Starting Echo Server..."
erl -boot echo-1.0 -pa /usr/local/Echo-1.0 -detached

If you’re wondering, it looks like this script gets installed to: /etc/init.d/suse_studio_custom

I installed the ‘open-vm-tools’ package, as suggested (since I had specified that I wanted to generate a VMware virtual machine), and then set the machine to be built.

The appliance took a few minutes to build.

(Having requested a few more builds, it turns out that there’s a queueing system, so you may get lucky, or you may be in for a wait…)

Once the build was complete I was able to try it out in the browser:

test-drive

After downloading it, double-clicking the .vmx file booted the machine in VMware:

vmware

I was then able to test out the Erlang application by connecting with telnet:

echo

Alternative Approaches

An alternative to actually installing your application is to just setup an appliance that has Erlang installed. This would still ease the setup process of the system and would just require you to add your Erlang application in after installation.

If you wanted the latest version of the Erlang system, it might be possible to upload the latest Erlang release as an ‘overlay file’, as opposed to fetching it from the repository. The benefit of using the repository is that it could be set to update automatically.

]]> http://joefreeman.co.uk/blog/2009/08/deploying-an-erlang-appliance-with-suse-studio/feed/ 3 Estimating Pi with Monte Carlo Methods http://joefreeman.co.uk/blog/2009/07/estimating-pi-with-monte-carlo-methods/ http://joefreeman.co.uk/blog/2009/07/estimating-pi-with-monte-carlo-methods/#comments Thu, 16 Jul 2009 12:58:16 +0000 Joe http://joefreeman.co.uk/blog/?p=77 This article demonstrates how to use a Monte Carlo simulation to calculate a value for Pi. An example program is given, written in Erlang, which can be run in a parallel environment. As well as the theory behind the method, we will look at how random numbers can be generated in Erlang and how to time Erlang programs. We will also perform an informal analysis of the results.

Monte Carlo Simulations

A Monte Carlo method is the general name of an algorithm that uses a stochastic process to perform repeated random sampling on a problem. A statistical approach is used such that the numerical result to the problem can then be given by considering the number of ‘hits’ that are achieved with the sampling.

The accuracy of the result, and the confidence in it, will increase as the number of samples we take increases.

More details about Monte Carlo methods can be found on the Wikipedia article and in the article The Beginning of the Monte Carlo Method.

Estimating Pi

A nice example of the use of Monte Carlo methods involves estimating a value for Pi. Let us consider a circle (with a radius of one) which is within a square (with edges of length two). Refer to the following diagram:

Diagram

Now, by choosing random points within the square, we are able to calculate whether or not each point is within the circle or not. We can consider the analogy of throwing darts ‘randomly’ at a dart board.

Since we know the area of a circle with radius r is given by:

Area of a circle

And the area of a square with sides of length x is given by:

Area of a square

And in our example, r=1 and x=2 and we know that the ratio, R, of the area of the circle to the total area of the square is:

Ratio of areas

Therefore, if we can come up with this ratio by experimentation, we can use it to derive a value for Pi:

Estimation of Pi

Generating Random Numbers in Erlang

Our method requires that we generate pairs of random numbers to identify the x- and y-co-ordinates of random points within the square. We need to make sure that these numbers have a uniform distribution—so that there is an equal chance of hitting any point on the dart board.

Fortunately, Erlang has just such a function: random:uniform/0. It is essentially as simple as calling random:uniform() to generate a random float between 0.0 and 1.0. Details can be found in the manual page.

Seeding

A note at the bottom of the manual page states:

If a process calls uniform/0 or uniform/1 without setting a seed first, seed/0 is called automatically.

Interestingly, if a number of processes are spawned together, they will all produce the same list of ‘random’ numbers. This is because each process is seeding the random number generator with the same default values.

To avoid this, we can seed the random number generator ourselves. The manual recommends using the built-in-function now/0. We can achieve this by calling the following line from the newly spawned process:

apply(random, seed, tuple_to_list(now()))

The function now/0 will always return a unique value, so we can be confident that each process has a uniquely seeded random number generator. Incidentally, I am led to believe that now/0 will actually move ahead of the system time if called frequently enough.

Code Listing

-module(pi).
 
-export([start/2]).
 
%% Start processing on the specified number of processes for the specified
%% number of points. The number of points will be split equally amongst the
%% number of processes.
 
start(Processes, Points) ->
	PointsPerProcess = Points div Processes,
	start_processes(Processes, PointsPerProcess),
	wait_for_results(Processes, Points).
 
 
%% Spawn the required number of processes. Each process is passed the Pid of
%% the current process so that it can send the result back.
 
start_processes(0, _PointsPerProcess) ->
	io:format("Started all processes.~n");
 
start_processes(Processes, PointsPerProcess) ->
	Pid = self(),
	spawn(fun() -> process_points(PointsPerProcess, Pid) end),
	start_processes(Processes - 1, PointsPerProcess).
 
 
%% Wait for the results to come back from each of the processeses. The result
%% from each process comes back containing the count of 'hits'. A total sum
%% of hits is calculated, and then when all the results have come back, we
%% give a calculated value for Pi.
%% 
%% Note: this is heavily reliant on the spawned processes not crashing, and
%% returning their respective counts. (Otherwise we will be waiting forever).
 
wait_for_results(Processes, Points) ->
	wait_for_results(Processes, Points, 0).
 
wait_for_results(0, Points, Sum) ->
	Pi = (Sum / Points) * 4,
	io:format("Pi: ~w~n", [Pi]);
 
wait_for_results(Processes, Points, Sum) ->
	receive
		{result, Count} ->
			wait_for_results(Processes - 1, Points, Sum + Count)
	end.
 
 
%% Setup processing of the points. We seed the random number generator, count
%% the number of 'hits' in the simulation, and then send the result back to
%% the parent process.
 
process_points(Points, Pid) ->
	apply(random, seed, tuple_to_list(now())),
	Count = test_points(Points, 0),
	Pid ! {result, Count}.
 
 
%% Each point is chosen randomly and then tested to see whether it is within
%% the circle. If it is, it is added to the count. Finally, the count is
%% returned.
 
test_points(0, Count) -> Count;
 
test_points(Points, Count) ->
	X = (random:uniform() * 2) - 1,
	Y = (random:uniform() * 2) - 1,
	I = X*X + Y*Y,
	if
		I < 1 ->
			test_points(Points - 1, Count + 1);
		true ->
			test_points(Points - 1, Count)
	end.

Note: I’ve only just started learning Erlang. Any feedback on my (simple) code will be welcomed.

Timing an Erlang Program

In order to experiment with running the program on multiple processors, it would be useful to be able to time the execution of the program. There is a module in Erlang for just this purpose. The function to use is: timer:tc/3. The three parameters that it takes are the module, the function and a list of arguments (these three parameters are commonly referred to as the MFA—module, function, arguments).

The following two lines allow us to time the program for ten thousand points running on one process and two processes, respectively:

timer:tc(pi, start, [1, 10000]).
timer:tc(pi, start, [2, 10000]).

The timer:tc/3 function returns the tuple {Time, Value}, where Time is the time that the function took to execute, in microseconds, and Value is the return value from the function we are timing.

Analysis of Results

I’m currently running this code on my MacBook, so as a result of having two CPU cores, running with any more than two processes doesn’t give any speed benefits. It is nice to see, however, that running with two processes does give a significant speed increase over just a single process:

One Processor Two Processors Speed-up
1 000 000 points 1.371 0.6980 1.964
10 000 000 points 13.44 6.854 1.961

Times above (and below) are averaged from three runs of each scenario and given in seconds.

Let’s now look at the estimations given for Pi:

Number of Points Time taken (seconds) Estimation for Pi
1 000 000 0.69800 3.14225733
10 000 000 6.8543 3.1418392
100 000 000 68.452 3.14162744
1 000 000 000 843.74 3.14161595

We can observe the convergence to the true value of Pi on a graph:

Convergence to Pi

The blue line shows our estimation of Pi, and the red line shows the true value of Pi. Note that the x-axis has a logarithmic scale.

Extensions

A nice extension to my program would be to be able to leave it running, and be able to call a function to see what the current progress is. For example, the number of points tested, and the current estimation of Pi would be given.

It would also be interesting to see how the program behaved on more than two processors. Specifically, when the processors are distributed over a network. I don’t currently have any other machines set up to try this out on.

]]> http://joefreeman.co.uk/blog/2009/07/estimating-pi-with-monte-carlo-methods/feed/ 0