Joe Freeman's Weblog

Handling Multipart Uploads with MochiWeb

Joe — Sat, 12 Dec 2009 00:02:40 +0000

I was getting along fairly well with my own custom-built, lightweight Erlang web server, until I needed to handle file uploading. At this point, I decided it was time to stop trying to re-invent the wheel (or take the lazy approach—whichever way you choose to look at it), and take another look at MochiWeb, which implements this feature.

However, MochiWeb’s documentation is non-existent (unless I’m missing something—please let me know if I am). I came across James Gardner’s post on handling file uploads, which helped me out. But after digging through the MochiWeb source-code I discovered a slightly easier way of handling file uploads. It may be that this method has been added since James’ post in 2007, or maybe it’s just less appropriate to James’ requirements.

I’m going to quickly run over how to setup MochiWeb. I know this has been covered quite a lot by other people, but it’s something a found a little daunting when I started out, so I want to emphasise that it’s a lot easier to setup than maybe first appears. Then I’ll post and explain a short chunk of code for handling uploads, and saving them to /tmp. Finally, I’ll provide an example of using this technique to provide a very simple photo gallery system.

Setting up MochiWeb

There’s a pretty good tutorial on the BeeBole blog, but I’ll cover the basics here.

First step is to check out the code from the Google Code repository. Make sure you have subversion installed, and then:

svn checkout http://mochiweb.googlecode.com/svn/trunk/ mochiweb

Now, make sure that you have Erlang installed, and:

cd mochiweb
make
chmod +x scripts/new_mochiweb.erl
./scripts/new_mochiweb.erl mochiweb_uploads ../.

This will build the MochiWeb system and then create a new MochiWeb project. You can repeat this process whenever you’re starting a new project. Note that mochiweb_uploads is the project’s name. MochiWeb is a bit picky about project names: they must be valid module names, so (it seems) you can’t use hyphens.

Next, we should make sure that the project builds and runs before we start coding:

cd ../mochiweb_uploads
make
./start-dev.sh

By default, your MochiWeb project is setup as a lightweight web server that serves files from the priv/www directory on port 8000. All being well, you should now be able to point your browser at http://localhost:8000/ and see the ‘MochiWeb is running.’ message.

If you have any problems, you’ll need to wade through the progress reports on the console to try and figure out where things are going wrong. The clues will be in any ‘crash reports’.

You can stop the server with Ctrl+C, then entering ‘a’, and hitting return. Or you can just hit Ctrl+C twice.

Handling File Uploads

First of all, we need to put together a page that we can test our uploading with. We’ll just replace the index.html file in priv/www with something straight-forward:

<html>
<head>
<title>MochiWeb Upload Test</title>
</head>
<body>
<form action="upload_photo" method="post" enctype="multipart/form-data">
    <input type="file" name="photo" />
    <input type="submit" value="Upload" />
</form>
</body>
</html>

Routing the Request

The code for starting the web server is in src/mochiweb_uploads_web.erl. Open up this file and take a look at the loop/2 function. The function is split up into two parts by the outer case construct: the first part is for HTTP GET (and HEAD) requests, the second is for POST requests. We’re going to be handling the file uploads in the POST clause. Like so:

'POST' ->
    case Path of
        "upload_photo" ->
            upload_photo(Req);
        _ ->
            Req:not_found()
    end;

All we are doing here, is delegating responsibility for handling the ‘upload_photo’ request to a function called ‘upload_photo’, which we can add to the bottom of mochiweb_uploads_web.erl:

upload_photo(Req) ->
    Req:ok({"text/html", [], "Hello, world!
"}).

That’s going to just return a ‘Hello, world!’ message so we can check the request is getting routed correctly. If we try out the system (make, ./start-dev.sh, go to localhost:8000), we should initially see the form, then after submitting the form our message should appear.

Handling the POSTed multipart data

Now onto actually handling the POSTed multipart data. We’ll replace our upload_photo/1 function with this:

upload_photo(Req) ->
    FileHandler = fun(Filename, ContentType) -> handle_file(Filename, ContentType) end,
    Files = mochiweb_multipart:parse_form(Req, FileHandler),
    Photo = proplists:get_value("photo", Files),
    % TODO: handle the photo here
    Req:ok({"text/html", [], "Thank you. \"index.html\">Upload another?
"}).

Let’s examine this line-by-line.

First of all, we’re specifying a function that we pass to the mochiweb_multipart:parse_form/2 function. The function that we’re passing will get called once for every file that is present in the POST data (i.e., that is in the form)—don’t forget that multiple files may be being uploaded.

The file will be split into ‘chunks’. So it’s the job of this ‘FileHandler’ function to return another function that will be used to consume each chunk of the file (and then, finally, the ‘eof’ atom). We’ll come back to this in a moment.

The parse_form/2 function will then return a list (in fact, a property list) of all the files. There will be a mapping from the name of the input to the value finally returned from our file handler.

Let’s take a look at the handle_file/2 function:

handle_file(Filename, ContentType) ->
    TempFilename = "/tmp/" ++ atom_to_list(?MODULE) ++ integer_to_list(erlang:phash2(make_ref())),
    {ok, File} = file:open(TempFilename, [raw, write]),
    chunk_handler(Filename, ContentType, TempFilename, File).

We use erlang:phash2/2 and make_ref/0 together with a (hopefully) application-specific prefix to construct a random filename which will reside in ‘/tmp’. A couple of points to make here: collisions aren’t impossible and this is platform-dependent. We could get the temporary directory from the operating system, and we could generate better filenames.

We open a file for writing to and pass it to our ‘chunk handler’ ready for consuming the first chunk. Note that chunk_handler/4 is a function that returns a function. We return this generated function back to the MochiWeb code, which will use it to handle the first chunk.

Before I go on to explain the ‘chunk_handler’, I should point out that in theory we could use this opportunity to filter the input. If (for some reason) we wanted to handle different types of file in different ways, we could do this based on the ContentType. Annoyingly, there’s no way to inspect the name of the field (i.e., the name of the HTML input) that we’re handling at this stage.

As I mentioned, the chunk_handler/4 function simply returns another function (which will accept one parameter) that MochiWeb will call once it has fetched a chunk. I’m using the term ‘chunk’ casually: it can be either a raw chunk of data from the socket, or the eof atom. If the chunk being passed is not the eof atom, the function must return another function which will be used to handle the next chunk. However, if the function is passed the eof atom, we will return a value that we want to associate with this part of the multipart data. The value returned here will be the value that is stored in the property list we mentioned earlier (in upload_photo/1).

Here’s our chunk_handler/4:

chunk_handler(Filename, ContentType, TempFilename, File) ->
    fun(Next) ->
        case Next of
            eof ->
                file:close(File),
                {Filename, ContentType, TempFilename};
            Data ->
                file:write(File, Data),
                chunk_handler(Filename, ContentType, TempFilename, File)
        end
    end.

So, don’t forget we’re returning a function here, which will get executed later on. There are two cases, as we’ve discussed:

The eof case involves closing the file, and returning a tuple containing the original filename, the content-type and the temporary filename. We could return this in record form, or we may choose just to return the temporary filename if we didn’t care about the other parameters.
The non-eof case involves writing the chunk of data to the file, and then returning a function to handle the next chunk of data.

This concludes the data-handling code. At this point you may wish to refer back to the upload_photo/1—this is where you would then be able to reference each of the uploaded files in the property list by it’s HTML input’s name. The value in the property list would be the {Filename, ContentType, TempFilename} tuple being returned by chunk_handler/4.

A Semi-practical Example

To turn this into something ‘useful’, I’ve put together a rudimentary photo gallery system.

We will adapt our upload_photo/1 function to take another two parameters (the directory for the photos and a list of valid file extensions; the function will hence become upload_photo/3).

The total of our security features will be to check that the extension of the file corresponds to a recognised image type. Anyone will be able to upload files.

Here’s our function:

upload_photo(Req, PhotoDir, ValidExtensions) ->
    FileHandler = fun(Filename, ContentType) -> handle_file(Filename, ContentType) end,
    Files = mochiweb_multipart:parse_form(Req, FileHandler),
    {OriginalFilename, _, TempFilename} = proplists:get_value("photo", Files),
    case lists:member(filename:extension(OriginalFilename), ValidExtensions) of
        true ->
            Destination = PhotoDir ++ OriginalFilename,
            case file:rename(TempFilename, Destination) of
                ok ->
                    Url = "/",
                    Req:respond({302, [{"Location", Url}], "Redirecting to " ++ Url});
                {error, Reason} ->
                    file:delete(TempFilename),
                    html_response(Req, "An error occured whilst trying to move your file: " ++ atom_to_list(Reason) ++ ". Does the destination directory exist?")
            end;
        false ->
            file:delete(TempFilename),
            html_response(Req, "Invalid file type. File extension must be one of: " ++ string:join(ValidExtensions, ", ") ++ ". \"/\">Try again?")
    end.

You can download the rest of the code. It’s not very elegant, but I was keen to put together something for you to take home.

Obviously this isn’t the sort of thing you should consider making available to the world wide web, but hopefully it’ll be useful in explaining how to handle file uploads.

Book Review: Ext JS 3.0 Cookbook

Joe — Wed, 25 Nov 2009 20:08:05 +0000

Packt Publishing contacted me recently (it would seem, among many others) to ask if I’d like to review one of their new books: Ext JS 3.0 Cookbook, by Jorge Ramon. I was more than happy to oblige, since the book looked like it would contain some interesting material.

The Book’s Structure

The book comprises “109 great recipes for building impressive rich internet applications using the Ext JS JavaScript library” (according to the front-cover—although I notice that the slogan on the picture above is different to my copy).

Each recipe starts with an introduction to the problem, and then a ‘How to do it…’ section. This is usually followed by a ‘How it works…’ section—where Jorge goes a little bit deeper into the inner workings of the Ext JS library. This is, I think, where the book shines. At lot of the examples in the book don’t seem to be any more special than the samples provided by ExtJS, but this added depth and explanation gives the examples more value than the ExtJS samples.

The book is split into ten chapters:

DOM and Data Types, the Ext JS Way
Laying Out a Rich User Interface
Load, Validate, and Submit Forms
Fun with Combo Boxes and Date Fields
Using Grid Panels to Display and Edit Tabular Data
More Applications of Grid and List Views
Keeping Tabs on Your Trees
Making Progress with Menus and Toolbars
Well-charted Territory
Patterns in Ext JS

The structure of the book is a little strange. It works fine as a reference book, but when reading from cover to cover, the concepts in the recipes rarely seem to flow from one to the next. For this reason, I think the book may not be ideal for beginners. It also means that some explanations get repeated in different recipes. However, the reference style of the book is benefitted by the ‘See also…’ section at the end of each recipe, which directs you to related recipes.

The book would have gained, I think, from an initial chapter introducing Ext JS, rather than jumping straight into methods for detecting the user’s browser—which isn’t a particularly interesting topic to start on anyway.

Code Samples

The book is largely made up of code samples, as you would expect. However, I have a few small criticisms to make of the code:

I feel that some of the samples are longer than they need to be (often a few pages long—the portlets recipe consists of nine pages of code interspersed with a few comments).
The choice of variable names is sometimes poor (although this may be following from the Ext JS convention).
Occasionally the formatting of the code is unhelpful—I noticed a comment had been wrapped, and on some examples the indentation was inconsistent.
Some of the code lines are highlighted in bold to make them stand out, but I found that quite often this was over-used, or highlighted the wrong areas—so turned out to be not so helpful.

This is nit-picking really though—the code is generally quite appropriate to the topics in question, and if you’re familiar with Ext JS, you shouldn’t have problems understanding the code—but I’d have expected more attention to these areas by a publishing company that has probably never published a book without source code.

Screenshots

Some of the screenshots are not very clear and the annotations are sometimes a little confusing. In one case, the same half-page screenshot seemed to get duplicated four times, with very minor changes between them.

A little colour and better annotations would have really benefitted the book, especially since it is largely focussed on user interface design (and RRPs at £30.99/$49.99!)

Further Observations

I was surprised to see the occasional use of PHP, which didn’t actually seem to be required to explain the topic. Giving a snippet of raw JSON may have been more appropriate.

I’m not sure how other books from Packt are structured, but I think the book would have benefitted from ’side boxes’ in the margin giving little extra hints, or explaining certain parts of the code in more detail. In my experience, this is a common and useful feature of computing books.

Conclusion

It was very nice to see a chapter on the use of design patterns in Ext JS. This is a topic that I found difficult when I started working with the library, and I didn’t manage to find much helpful material online at the time. The recipe on creating customised ‘vtypes’ was also interesting.

I feel like I’ve come up with plenty of criticisms, but in general I think the book is worth reading, even if just for the ~50% original material (which isn’t already available on the Ext JS samples page), and for the brief-but-valuable explanations of how things are working ‘under the hood’.

As a reference book, it feels very complete. I haven’t been able to think of any topics that it misses out. Mixing the recipes together should result in a substantial meal, and some very satisfied clients.

Thank you Packt for sending me the book, and thanks Jorge for putting so many useful examples together in one resource.

Update: I requested a link to some sample content: here’s the whole of Chapter 3, Load, Validate, and Submit Forms, for you to download.

A Java Swing Application for Testing PHP Snippets

Joe — Mon, 09 Nov 2009 16:34:44 +0000

I occasionally find myself needing to try out short chunks of PHP code. For example, to try out a regular expression, or to see how a built-in function behaves if it’s not documented very well. Creating a file, moving it to my web server’s document root, loading it up in the browser often seems too much trouble. So I’ve put together a simple application using Swing to make this easier.

Overview of the Application

Let me summarise the functionality of the application, incase you just want to try it out. The app allows you to enter PHP code and then execute it without having to save it to file. You see the results instantly in the bottom half of the window.

There is also a built-in, simple (single-threaded) web server. This means you can view the results of execution in your browser—again, without having to save the file.

I’m going to explain some of the technical aspects of putting the app together. If you’re just interested in using the app, scroll down to the bottom of the post for the download link.

Working with the Command Line from Java

PHP can be run from the command line. Details about this are on the PHP website, but in short, if you run php with no arguments, then it will take in source code from the standard input, execute it, and print out the result.

This can be achieve programatically from within the Java environment:

// Get the runtime object
Runtime rt = Runtime.getRuntime();
 
// Run the command
Process process = rt.exec(PHP_COMMAND);
 
// Setup buffered input and output streams
BufferedReader input = new BufferedReader(new InputStreamReader(process.getInputStream()));
BufferedWriter output = new BufferedWriter(new OutputStreamWriter(process.getOutputStream()));
 
// Send the source code and close the stream (to make EOF)
output.write(code);
output.close();
 
// Read in the result, line by line and print it to the console
String line;
while((line=input.readLine()) != null) {
    System.out.println(line);
}
 
// Wait for the process to terminate (not really necessary, because the process will terminate once it has closed the input stream anyway)
int result = process.waitFor();

Note that rt.exec(...) and process.waitFor() can both potentially throw exceptions—IOException and InterruptedException respectively—which must be dealt with.

We could get the standard error stream aswell—using the same approach as with the output stream—but PHP doesn’t send anything to the error stream, so we don’t need to worry about it. If we wanted to do this, we’d need to read both streams in separate threads.

Building the Swing User Interface

Let’s look at how to setup the user interface. I’ll talk through a simplified version of the final application. We’ll look at a few of the Swing components individually.

`JFrame`

We’re going to make use of the JFrame class as our base. A common practice is to extend this class with your own, however, generally a more practical approach is to encapsulate the JFrame object by keeping track of the JFrame as a property within your class, and exposing methods as necessary.

We create a JFrame, set a layout (refer to the next section), a preferred size, the behaviour to associate with the window’s ‘close’ button, pack it (to make sure the components get arranged properly) and then finally make the frame visible to the user:

frame = new JFrame(FRAME_TITLE);
 
frame.setLayout(new BorderLayout());
frame.setPreferredSize(new Dimension(500, 500));
 
frame.setDefaultCloseOperation(JFrame.EXIT_ON_CLOSE);
 
// Add components to the frame here
 
frame.pack();
frame.setVisible(true);

`BorderLayout`

Once the JFrame has been setup, we’ll use a BorderLayout to arrange our components on the window. Border layouts are a fairly common concept in user interface design. One component can be attached to each of five regions. Each region defines the position that the component appears at, and how its size changes as the parent component is resized. Border layouts can be nested within each other. The regions are:

PAGE_START – generally at the top of the page, resizes to fill the width of the parent component.
PAGE_END – opposite side to PAGE_START.
LINE_START – generally on the left of the page (but actually dependent on the user’s language), resizes to fill the height of the parent minus the height of any components at PAGE_START and PAGE_END.
LINE_END – opposite side to LINE_END.
CENTER – the remaining space in the middle of the layout, resizes to fill the available space.

See the example below, which shows the positioning of the components:

`JSplitPane`

For our user interface we will divide the window into two: the top half for editing the PHP code, and the bottom half for viewing the results of the execution. We will allow the user to move the separator between the two components using a JSplitPane.

Using a JSplitPane is fairly straight forward. Pass each of the two components you’re splitting into the constructor, along with a specified orientation (either VERTICAL_SPLIT or HORIZONTAL_SPLIT). The location of the divider can also be given.

JSplitPane splitPane = new JSplitPane(JSplitPane.VERTICAL_SPLIT, inputPanel, outputPanel);
splitPane.setDividerLocation(200);

`JTextArea` and `JScrollPane`

Both the editor and the area for displaying execution results will be based on JTextAreas. On their own, these components don’t support scrolling (e.g., for when there are more lines of text than will fit in the component). To support scrolling, we need to wrap them with JScrollPanes.

JScrollPane inputScrollPane = new JScrollPane(inputBox);

`JButton` and `Action`s

When the user wants to execute their code, they will click on an ‘Execute’ button (which will be a JButton). There are a few ways to handle button clicks in Swing. One such method is to define an Action that will be associated with the button. This is a nice way to separate logic from the design of the user interface.

A button can be created like this:

JButton executeButton = new JButton(new ExecuteAction(inputBox, outputBox));

The easiest way to create the action is to extend from AbstractAction:

public class ExecuteAction extends AbstractAction {
 
    private JTextArea inputBox;
    private JTextArea outputBox;
 
    public ExecuteAction(JTextArea inputBox, JTextArea outputBox) {
 
        // Save these components for later
        this.inputBox = inputBox;
        this.outputBox = outputBox;
 
        // Set the name of the button
        putValue(AbstractAction.NAME, "Execute");
    }
 
    public void actionPerformed(ActionEvent e) {
        // Event will be handled here
    }
}

Any objects that the action needs to access get passed to the constructor, and stored as properties of the action for use later. In this case, we’ll want to be able to get the source code from the input box and then put the result into the output box.

Putting Together the `PhpRunner` Class

We will put our earlier PHP-running code into it’s own class and package (runner.PhpRunner). The class will be constructed with two parameters:

The PHP code to be run
An event listener (an object implementing the OutputConsumer interface)

The execute() method can then be called. This will start the execution, and notify the event listener every time we get a new line of output back. This approach means that if a PHP script takes a long time to complete, we’ll be able to see the output as it happens, rather than having to wait for the whole script to finish executing.

The code is similar to the code at the top of the post, so I won’t paste it again. I have added calls to the event listener though. Refer to the source code if you’re interested.

Threads and Thread Safety

Introducing threads into a Swing application can be dangerous, since Swing’s components are not thread safe. Only one thread should be allowed to access the Swing components at any one time—this is known as the event-dispatching thread (EDT).

However, we don’t want the user interface to freeze whilst the PHP code is being executed, so it’s important that we don’t wait for output from within the EDT. The swing library provides a SwingWorker class, which makes working with threads a bit easier and safer. Fortunately, our PhpRunner fits nicely into this pattern. A cut-down version of the code looks like this:

new SwingWorker<String, String>() {
 
    @Override
    protected String doInBackground() throws Exception {
        // Setup the PHP runner, and listen for any lines of output
        PhpRunner runner = new PhpRunner(code, new OutputConsumer() {
            @Override
            public void output(String line) {
                // Let the SwingWorker know there are more lines
                publish(line);
            }
        });
        // Execute, and return the result
        return runner.execute();
    }
 
    @Override
    protected void process(List<String> chunks) {
        // Add any output to the output box
        for (String chunk : chunks) {
            outputBox.append(chunk + "\n");
        }
    }
 
    @Override
    protected void done() {
        // The operation has completed
    }
}.execute();

Adding a Built-in Web Server

Sometimes it’s useful to be able to test the PHP output in a web browser. So we will add a very basic web server to the application. The user will be able to request execution of the code by loading a URL in their web browser.

First of all, we need to add a button to the user interface to allow the user to manually start and stop the web server. When the associated action is performed, we will either create a new HTTP server object, or stop an existing one.

The HTTP server doesn’t need to be very complicated at all. We don’t have to handle POST requests (just GET and HEAD requests) or even URLs since all requests to the server will execute the same script. We also only need to handle one request at a time—so the server can happily run on a single thread.

I don’t want to go into details of the server implementation here, but it’s worth nothing that this use of a web server does bring up the slightly unconventional approach of trying to get the contents of a Swing component from outside the EDT:

final String[] value = {""};
try {
    SwingUtilities.invokeAndWait(new Runnable() {
        public void run() {
            value[0] = inputBox.getText();
        }
    });
} catch (Exception e) {
    // Do nothing
}
return value[0];

The value variable must be declared final, since it’s accessed from a different thread. However, we want to be able to update the value, so the use of a single-element array is a bit of a hack to bypass the compiler’s warnings. A more elegant solution to this might involve wrapping the method in it’s own class, and using a class property instead of a final variable.

Saving User Settings with the Preferences API

Java provides an API that makes it very straight-forward to store and retrieve user preferences. These are used in our application to save a number of different parameters:

The position and location of the window on the screen.
The position of the resizable split pane.
The last-used PHP code.
The specified font size.
The directory of the last file that was loaded.

We can wrap the Preferences class in our own Settings class, which provides methods to get the required properties. Attaching a WindowListener to our JFrame means that we can attempt to save the settings when the application is closed.

Conclusions

I ended up making a few more changes to the application to those mentioned above. The buttons that are referred to above have been move to the menu bar (which was easy to do, since I was using Actions), and have associated keyboard shortcuts which make the application much easier to use. It’s now also possible to load files from disk, change the font size of the input and output boxes. I’ve also added a status bar which shows status messages for a period of time, and a ‘preferences’ dialog which allows you to change the PHP command path and the web server port.

If the implementation of any of these features interests you, I suggest you download the source code to have a look.

To get a grasp of the structure of the project, the layout below may help:

I’m very pleased with the final product. The application’s concept is simple, but I think it adheres well to the “do one thing and do it well” philosophy.

I’ve also made a few minor tweaks to the application that will affect Mac users: moving the menu bar to the top of the screen, and setting the application’s title.

Download

If you’d like to download the application to give it a try yourself, or if you’d like to download the source code in its entirety, use the links below:

If you do try out the application, please let me know what you think.

Requirements

The application requires that you have both Java (version 1.6—although 1.5 might work) and PHP installed. The PHP executable must be accessible on your ‘path’. I have tested this on OS X and on a vanilla install of Vista (after installing Java and PHP).

FULLTEXT Search with MySQL and CodeIgniter

Joe — Mon, 26 Oct 2009 10:17:27 +0000

I was required to use CodeIgniter for a project recently. I’m generally not a big fan of the framework, but I have grown to like aspects of it. A lot of people seem to praise the documentation, the mass of examples and the active community. Maybe my hopes were set too high, but I was disappointed with all of these.

The thing I like about using a framework is being told how to do things, but for many the big attraction to CodeIgniter seems to be that it’s lightweight and you can do things how you like. I think this is the reason for the inconsistencies and low quality of example code. So I thought that I’d put together an article and example project showing a few of the tricks I’ve learnt, and my preferred CodeIgniter style.

I’m going to put together a simple application which shows off the search capabilities of MySQL, whilst exploring some of CodeIgniter’s features.

Getting CodeIgniter Ready

This is pretty basic stuff. Download, unzip, and dump the files somewhere into your web server’s directory. Setup a MySQL database and associated user and fill in the connection details in application/config/database.php.

Open up application/config/config.php and edit the base_url setting. This needs to be the web-visible URL to the directory that you put the CodeIgniter files into. If you did put them into your local web server’s root directory, this may be as simple as ‘http://localhost/’.

Setting Up the Database

You should already have a database setup, but we need a table to store the data. I’ve created a simple ‘pages’ table. To make use of MySQL’s FULLTEXT search capabilities, you must setup an index on any fields that you want to be searchable. See the the SQL below:

CREATE TABLE pages (
  id int(10) UNSIGNED NOT NULL AUTO_INCREMENT,
  url text NOT NULL,
  title text NOT NULL,
  content text NOT NULL,
  updated datetime NOT NULL,
  PRIMARY KEY  (id),
  FULLTEXT KEY content (content)
) ENGINE=MyISAM

I’ve setup an index on the ‘content’ field so that it will be searchable. Note also that FULLTEXT search is only available when using the MyISAM engine.

Some Test Data

In order to perform a decent evaluation of the searching mechanism, I realised that I would need some substantial data. I decided Wikipedia would make a good source of this data. Rather than export all three million articles, I limited myself to about two thousand of the featured articles. I setup a script to scrape the list of featured articles and then used the Wikipedia export feature to automatically copy across all the data into my database.

After over-coming an issue with the character encoding it soon because obvious that the Wikipedia markup wasn’t quite the format that I wanted. Hacking together a few regular expressions meant I could strip most of this out, leaving plain and readable content.

Searching the Data

At this point, the magic SQL to use takes the form: MATCH (columns) AGAINST ('terms'). Note that when MATCH ... AGAINST is used in the WHERE clause, the results are automatically sorted, so there’s no need to add an ORDER BY clause. An example:

SELECT *
FROM pages
WHERE MATCH (content) AGAINST ('test') > 0

There are a few FULLTEXT gotchas you should be aware of:

Something that I was stung by: if you have fewer than three rows of data, you won’t get any search results back.
Search strings have a minimum length. This is generally three or four characters, but depends on configuration. See below.
There is a collection of ‘stopwords‘ that MySQL will ignore (words like ‘the’, ‘however’, ‘hello’). If you try a search for any of these, you won’t get any results back.

You can find out some of the configuration details by executing SHOW VARIABLES LIKE 'ft%'. You should get something like this:

A few different ‘modes’ are available for searching. By default, a natural language search is performed. A boolean search is also possible, and allows a number of powerful operators to be used – refer to the documentation for details.

It is also possible to use query expansion. This roughly means that the search may return items where the search terms aren’t actually in the content, but instead there are words that are deemed similar. This is achieved by performing two searches. Where the second search uses content from high-ranking rows in the first search to perform the second search. This only tends to work well for short queries, otherwise less-relevant results will be returned.

More details on the full-text functionality can be found in the MySQL documentation.

CodeIgniter and the MVC Pattern

Let us return our attention to CodeIgniter…

The framework takes advantage of the ever-popular model-view-controller (MVC) pattern. Everyone knows you’re supposed to separate your domain-logic from presentation code, but the nice thing about the MVC pattern is that it’s constantly there to remind you.

I stick to a few basic rules:

you can only touch the database from within the model
nothing should be echo-ed from the controller (or the model)
only one statement can appear inside each set of alternative syntax)

If you find yourself breaking any of these rules, it’s probably a sign that it’s time to step back and re-think what you’re trying to achieve.

Let’s start off with the model. I only have one method here, which performs the searching. You may note that I’m using raw SQL here rather than using CodeIgniter’s Active Record implementation. This is just a personal preference – I think the CodeIgniter implementation is a bit messy compared to something more RoR-like.

The following code needs to go into application/models/page_model.php:

class Page_model extends Model
{
    function Page_model()
    {
        parent::Model();
 
        // Make the database available to all the methods
        $this->load->database();
    }
 
    function search($terms)
    {
        // Execute our SQL statement and return the result
        $sql = "SELECT url, title
                    FROM pages
                    WHERE MATCH (content) AGAINST (?) > 0";
        $query = $this->db->query($sql, array($terms, $terms));
        return $query->result();
    }
}

Next up is the view. Note that I’m religiously using the form helpers. This needs to go into application/views/search_results.php:

 $this->load->helper('form'); ?>
 
 echo form_open($this->uri->uri_string); ?>
 echo form_label('Search:', 'search-box'); ?>
 echo form_input(array('name' => 'q', 'id' => 'search-box', 'value' => $search_terms)); ?>
 echo form_submit('search', 'Search'); ?>
 echo form_close(); ?>
 
 if ( ! is_null($results)): ?>
     if (count($results)): ?>
        
         foreach ($results as $result): ?>
             echo $result->url; ?>"> echo $result->title; ?>
         endforeach ?>
        
     else: ?>
        There are no results for your query.
     endif ?>
 endif ?>

Finally, pulling both the model and view together with the controller. The following code should go into application/controllers/pages.php:

class Pages extends Controller {
 
    function search($search_terms = '')
    {
        // If the form has been submitted, rewrite the URL so that the search
        // terms can be passed as a parameter to the action. Note that there
        // are some issues with certain characters here.
        if ($this->input->post('q'))
        {
            redirect('/pages/search/' . $this->input->post('q'));
        }
 
        if ($search_terms)
        {
            // Load the model and perform the search
            $this->load->model('page_model');
            $results = $this->page_model->search($search_terms);
        }
 
        // Render the view, passing it the necessary data
        $this->load->view('search_results', array(
            'search_terms' => $search_terms,
            'results' => @$results
        ));
    }
}

At the beginning of the method, there is a reference to a redirect function. This is a helper function, and it’s part of the ‘url’ group of helper functions. To use it, we could load the helper manually using: $this->load->helper('url');. However, because this group of helper functions is quite commonly used, we can load them automatically. To do this, we need to edit the application/config/autoload.php file:

$autoload['helper'] = array('url');

We should now have a working prototype. Make sure you have some data in your database (remember, at least three rows). Then steer your browser to the relevant URL, which should be something like: ‘http://localhost/index.php/pages/search’, enter some text to search (something that you know is in the database) and hit the ‘Search’ button.

You should get something like this:

Adding More Features

To show off a bit more of CodeIgniter, I’ll add a few more features, including pagination, custom configuration files and custom helpers.

The CodeIgniter user guide boasts that the pagination class is “100% customizable”. I wouldn’t go that far, but you can certainly change a few things.

First of all, we need to modify our model so that it only returns a portion of the results, and also so that we can find out how many results in total our query would return.

The first problem is easily solved by adding an argument to the method, and a LIMIT to our SQL query. For the second problem, there are a few solutions, none of which seem very elegant:

We can add a separate method to the model, which will return the total number of results. This means that generally we will always have to call both routines. Certainly not a disaster, but it just doesn’t seem very refined.
Alternatively, we can calculate this count from within our ’search’ method, and return both a collection of results and the total count together in some form of array structure. This is a bit abusive of the only-return-one thing methodology.
Or, we can pass a reference parameter to the method, set the value inside the method, and then access this variable from the controller. I think this works quite nicely because it’s discrete, but possibly a bit too subtle to be obvious to other developers.

To keep things simple, I’m going to go with the first option. Let’s refactor the model:

class Page_model extends Model {
 
    // Constructor as before
 
    function search($terms, $start = 0, $results_per_page = 0)
    {
        // Determine whether we need to limit the results
        if ($results_per_page > 0)
        {
            $limit = "LIMIT $start, $results_per_page";
        }
        else
        {
            $limit = '';
        }
 
        // Execute our SQL statement and return the result
        $sql = "SELECT url, title, content
                    FROM pages
                    WHERE MATCH (content) AGAINST (?) > 0
                    $limit";
        $query = $this->db->query($sql, array($terms, $terms));
        return $query->result();
    }
 
    function count_search_results($terms)
    {
        // Run SQL to count the total number of search results
        $sql = "SELECT COUNT(*) AS count
                    FROM pages
                    WHERE MATCH (content) AGAINST (?)";
        $query = $this->db->query($sql, array($terms));
        return $query->row()->count;
    }
}

Note that I’m also returning the ‘content’ field in the SQL – this will be useful later. The next thing to do is to modify our controller. We need to load and initialise the pagination library. Here’s the modified ’search’ method:

class Pages extends Controller {
 
    function search($search_terms = '', $start = 0)
    {
        // If the form has been submitted, rewrite the URL so that the search
        // terms can be passed as a parameter to the action. Note that there
        // are some issues with certain characters here.
        if ($this->input->post('q'))
        {
            redirect('/pages/search/' . $this->input->post('q'));
        }
 
        if ($search_terms)
        {
            // Determine the number of results to display per page
            $results_per_page = $this->config->item('results_per_page');
 
            // Load the model, perform the search and establish the total
            // number of results
            $this->load->model('page_model');
            $results = $this->page_model->search($search_terms, $start, $results_per_page);
            $total_results = $this->page_model->count_search_results($search_terms);
 
            // Call a method to setup pagination
            $this->_setup_pagination('/pages/search/' . $search_terms . '/', $total_results, $results_per_page);
 
            // Work out which results are being displayed
            $first_result = $start + 1;
            $last_result = min($start + $results_per_page, $total_results);
        }
 
        // Render the view, passing it the necessary data
        $this->load->view('search_results', array(
            'search_terms' => $search_terms,
            'first_result' => @$first_result,
            'last_result' => @$last_result,
            'total_results' => @$total_results,
            'results' => @$results
        ));
    }
 
    function _setup_pagination($url, $total_results, $results_per_page)
    {
        // Ensure the pagination library is loaded
        $this->load->library('pagination');
 
        // This is messy. I'm not sure why the pagination class can't work
        // this out itself...
        $uri_segment = count(explode('/', $url));
 
        // Initialise the pagination class, passing in some minimum parameters
        $this->pagination->initialize(array(
            'base_url' => site_url($url),
            'uri_segment' => $uri_segment,
            'total_rows' => $total_results,
            'per_page' => $results_per_page
        ));
    }
}

Let’s look at the changes we’ve made…

We have added an extra argument to the method. This means the URL will have another (optional) segment to it. It will be a number representing the record that we are starting on, and will get set automatically by the pagination code.

In order to determine the number of results to show on a page at a time, I have used the config library. In order to setup custom configuration for your application, it’s best to add a new file to the application/config directory, then set it to be autoloaded. Edit the application/config/autoload.php file again, and add ‘application’ to the $autoload['config'] array. Then create the application/config/application.php file, and put the following line in it:

$config['results_per_page'] = 10;

The next change we have made is to setup the pagination. I have put this code into a separate method. Note that the method name is prefixed with an underscore. This prevents the private method being called as an action on the controller. The code here is fairly self-explanatory.

We also pass some extra information over to our view. Which leads us to our next task of updating the view. But first, we will write our own custom helper functions for use within the view. These go into application/helpers/search_helper.php. The first function – search_highlight($text, $search_terms) – provides a way to highlight search terms in a string (there is a similar CodeIgniter helper, but it doesn’t quite do what we want). The second function – search_extract($content, $search_terms, $number_of_snippets = 3, $snippet_length = 60) – is a flaky attempt to generate an ‘excerpt’ to accompany each result. An excerpt is made up of a number of ’snippets’ which should each contain at least one of the search terms. I won’t paste the code here since it’s far from elegant, but it’s available in the download.

Finally, the updated view. Note that we have to manually load the helper before using the helper functions. The pagination links are created using the create_links method.

 $this->load->helper(array('form', 'search')); ?>
 
 echo form_open($this->uri->uri_string); ?>
 echo form_label('Search:', 'search-box'); ?>
 echo form_input(array('name' => 'q', 'id' => 'search-box', 'value' => $search_terms)); ?>
 echo form_submit('search', 'Search'); ?>
 echo form_close(); ?>
 
 if ( ! is_null($results)): ?>
     if (count($results)): ?>
 
        Showing search results for ' echo $search_terms; ?>' ( echo $first_result; ?>– echo $last_result; ?> of  echo $total_results; ?>):
 
        
         foreach ($results as $result): ?>
             echo $result->url; ?>"> echo search_highlight($result->title, $search_terms); ?>
 echo search_extract($result->content, $search_terms); ?>
         endforeach ?>
        
 
         echo $this->pagination->create_links(); ?>
 
     else: ?>
        There are no results for your query.
     endif ?>
 endif ?>

Here’s the result of trying it out in the browser again:

Evaluation

CodeIgniter comes with benchmarking and profiling features. We can mark points in the execution, enable profiling, and have them have them output to the browser. For example, I might add marks around the calls to the model:

// Mark the start of search
$this->benchmark->mark('search_start');
 
// Load the model, perform the search and establish the total
// number of results
$this->load->model('page_model');
$results = $this->page_model->search($search_terms, $start, $results_per_page);
$total_results = $this->page_model->count_search_results($search_terms);
 
// Mark the end of search
$this->benchmark->mark('search_end');

The output at the bottom of the page would look something like this:

We can see that the search on 1800 rows took 0.0138 seconds. This is 49% of the total execution time.

It is also possible to measure the time it takes to render the view by putting marks in the controller on either side of the $this->load->view(...) call. The view takes about 0.007 seconds to load. Without the generation of a page extract, it takes significantly less time: 0.002 seconds.

Download

I have packaged up the code for this example. Feel free to download it.

Lineographic Interpretations of Images, with an Etch-a-sketch

Joe — Tue, 29 Sep 2009 07:52:22 +0000

A significant part of my computing degree was the individual project. I’d like to take this opportunity to talk about what I did, and hence provide a home for the project online. I’m not going to provide too many concrete details of the implementation—just outline the concept, talk about some of the challenges, and comment on the results.

Selecting the Project

I decided to propose my own project, rather than adopt one of the staff proposals. The idea being that I could choose something ‘fun’ that I would retain interest in. The initial problem is coming up with an idea which covers the criteria of a good project:

it should require a notable amount of research, rather than just being a relatively straight-forward implementation problem
it should be challenging enough so that it provides enough scope to demonstrate your abilities, whilst not being too difficult

I came up with a few ideas, some of which I still have intentions of pursuing. But the idea that I finally settled on seemed like it had the best scope in terms of the above criteria.

During the period that I was trying to come up with a project idea, I came across a video of an etch-a-sketch artist. I don’t recall the exact video, but there are plenty online. I was amazed at the skill and patience of the artist. The sketches take days to complete, and it made me wonder if it was the sort of thing a computer could be made to do.

Essentially this leads to the question: is a computer better at re-creating images with an etch-a-sketch than a human? Various interpretations of the word ‘better’ can be taken, but I will be looking at both the aesthetic results and the time it takes to complete the sketch.

The process of drawing an image on an etch-a-sketch is essentially that of lineography—drawing the whole image without taking your pen off the paper.

Overview

The project takes an image as its input and uses several techniques to extract features in the image before attempting to find a single path that can be used to represent the image. The path is then serialised, and passed either to an etch-a-sketch simulator, or over Bluetooth to an external device connected to a real etch-a-sketch.

Feature Extraction

A number of techniques were experimented with for feature extraction. The most promising was based on segmentation of the image—specifically, by a region-growing technique, based on the intensity of pixels.

In order to reduce ‘noise’, segments with a size below a given threshold will be absorbed by neighbouring segments.

The next stage is to come up with a path around the edge of segments. To achieve this, a technique very similar to the ‘marching squares‘ algorithm is used. The following image shows a trivial example with four segments, with paths around the segments shown as dashed lines:

Once paths have been found, the problem can be represented as a graph, and hence becomes a path-finding exercise. The graph is made up of nodes that represent the points were more than two segments meet (marked as circles on the image above) and edges between the nodes are annotated with the corresponding paths in the image. The following image shows how a graph can be constructed from our previous example:

Path Finding

The path finding problem is quite unique. Features of the problem are shared with the travelling salesman problem in that we are aiming to traverse all the edges (although the TSP actually aims to visit all the nodes). However we aren’t searching for an optimal solution—instead we are looking for an aesthetically pleasing solution.

The edges between neighbouring segments that have significantly different colours (based on their Euclidean distance) are given a higher weight, and hence are more likely to be drawn over repeatedly. This results in a darker line in the finished sketch (refer to the images below).

The general technique is to traverse the graph with priority given to edges that we want to appear darker, and then back-tracing when we get stuck, to a point that allows us to continue to unused edges.

Occasionally ‘hops’ must be taken, where two points must be joined so as to connect unconnected sections of the graph. In these cases, various alternative paths will be investigated to attempt to minimise disruption to the sketch. This usually results in a pair of smaller hops to and from a neighbouring line.

Drawing Techniques

A few techniques for drawing were experimented with. These were inspired by work done by human etch-a-sketch artists. Eventually, a hybrid technique was settled on: first to draw the outline of features in the image, and then to shade in regions based on their intensity. Shading was achieved by hatching—drawing backwards and forth over the region.

The following two images show relatively successful results being displayed on the simulator:

See the image below for a screenshot of the ‘Sketcher’ application: part of the framework developed for testing the various algorithms—referred to as ‘engines’:

Hystereis

Extensive testing with a real-life etch-a-sketch device revealed that a significant error would occur when trying to control the device programatically.

When a knob is rotated first in one direction, and then in the other, it takes a certain amount of rotation in the second direction before the pen begins to move again on the screen. This is due to tension building up in the strings within the device. I have referred to the error that this effect introduces as the ‘hysteresis’.

The effect is taken into account in the etch-a-sketch simulator so that it can be correctly accounted for when a path is being finalised. I won’t go into details here, but will mention that trying to simulate this effect in two dimensions is significantly more complicated than I was anticipating.

Nxt-A-Sketch

An extension to the project was to connect an actual etch-a-sketch to the program and draw out the sketch using stepper motors. To achieve this, a Lego NXT brick was used. A small C program was written to listen for commands via a Bluetooth channel and control the stepper motors. The images below shows how this physical connection was made:

This extension was relatively successful, and certainly made the final presentation more engaging. The device was able to recreate the sketches at a recognisable level, but would take a while to draw (about an hour)—though still significantly less time than a human artist would take. However, mechanical problems were frequently encountered with the device (likely caused by the wires slipping within the etch-a-sketch), which can be observed in the following photograph:

Conclusion

At this stage, it is important to consider whether or not the project was successful. It is difficult to achieve this objectively, but it was certainly able to create sketches far better than the average human can, and sketches can be produced much quicker than those done by artists.

However, to answer this question in a slightly more analytical way, we can compare the results of the program with sketches done by artists. Below are two such comparisons. The first shows what I would consider a successful sketch of Mario; the second, a less successful interpretation of Da Vinci’s Mona Lisa (original images from here and here):

Clearly the project has room for improvement. A more intelligent method of feature detection is required, and it would be nice if the program had some form of artistic abilities.

Having said this, I like to think that the level of sketches being produced are better than I would be able to do had I spent the time that I devoted to the project instead practicing with the device.

Tests revealed that my accounting for hysteresis was particularly successful. However, I won’t go into details here, as the tests are rather complicated to explain.

Further Information

I have essentially attempted to squash a large portion of my report into this post, so a lot of the details have been glossed over or simplified. If anyone is interested in a specific area of the project, feel free to contact me for more information.

Deploying an Erlang Appliance with SUSE Studio

Joe — Tue, 04 Aug 2009 08:27:54 +0000

Having read Nat Friedman’s blog post about the release of SUSE Studio, I thought I’d take a look at it. I was also interested to find out how easy it is to create an appliance for a simple Erlang application.

An Introduction to SUSE Studio

I’m very impressed with SUSE Studio. It allows you to completely customise a Linux distribution and actually try it out (in the browser!) before you download it. This is, in my opinion, a pretty novel concept. At least at this scale.

Being able to have this control before you download the distribution obviously saves you from downloading a load of components that you don’t want. But the benefits don’t stop there:

Performing the administrative tasks of setting up the system can be performed before installation. This means a system can be deployed many times with minimum effort. Products like Xen and VMware provide cost-effective ways to host several different systems together and keep them isolated from each other. If one of your applications breaks, the others are (potentially) isolated from the problem.
Applications can be deployed and run in a pre-specified environment. This means their behaviour will be more predictable, and it will be easier to provide support for them. If a client is reporting problems, it will be much easier to replicate them yourself in a test environment.

My Experiences with SUSE Studio

The service is currently by invite only. But having filled out the form, I got an invite the next day:

Thank you for your interest in SUSE Studio, a new service that makes it quick and easy to create and test software appliances from your web browser. We are pleased to invite you to try SUSE Studio.

The interface is very smooth and pleasant to use. Helpful messages appear in the left-hand column, which let you know where you’ve gone wrong, or give you tips as you go, and there’s a helpful ‘undo’ feature. The site is also scattered with a selection of pleasing cartoons.

The ‘test drive’ feature is awesome. Once you’ve finished configuring the system you need to build it, which takes a little time. When this is completed, just click on ‘Testdrive’, and a flash applet lets you watch the machine and interact with it (via VNC). Nat has an article about some of the implementation details.

By enabling networking in test drive, you can also connect to the machine via SSH, or with your web browser if there’s a web server running.

Whilst Test Drive is running, you can even see a list of the files that have been modified, view diffs and download the modified files.

A impressive number of configuration parameters are at your disposal. You can even personalise the system by changing the logo and background image.

I noted a minor problem with the interface so I sent some feedback to the developers to check they were aware of it. I was impressed that they replied to me about ten minutes later (to say they were aware of the problem, thanks).

Setting up an Appliance with Erlang

The steps involved in setting up a basic appliance are very simple. I’ll skim over these steps and instead focus on those involved in setting up Erlang, and adding your application to the appliance. The aim is to have an Erlang application boot automatically when the appliance is started.

I started off with the openSUSE 11.1 ‘just enough’ base template, which, for reference, has the following packages installed:

aaa_base, branding-openSUSE, dhcpcd, grub, hwinfo, insserv, kbd, kernel-default, licenses, mkinitrd, module-init-tools, netcfg, openssh, openSUSE-build-key, openSUSE-release, PolicyKit, polkit-default-privs, procps, pwdutils, rpcbind, rpm, sysconfig, syslog-ng, vim

The next step is to add a repository that contains the Erlang package. Select ‘Add repositories…’ from the Software page, search for ‘erlang’ and then add the suggested ‘devel:languages:erlang 11.1′ repository. At time of writing, this will install Erlang release R12B-5.

Then, from the software page, search for ‘erlang’, and select the ‘erlang’ package to be added.

Then make a few more configuration changes as required—for example, selecting the keyboard layout and time zone, and adding users (with default passwords that should obviously be changed following installation).

The next step is to add the Erlang application. I won’t go into details of setting up OTP applications, but I will explain how to automatically generate a bootable release file.

The release file needs to be setup to work with the installed version of Erlang, and the boot script needs to be generated in the same system environment (so we know we’ve got the correct versions of applications). To achieve this, I have written an Erlang script to be run when the appliance is first booted, which will generate a release file and accompanying boot script.

The main function that generates the release file is:

generate_release(Name, Version, Apps) ->
    ErtsVsn = erlang:system_info(version),
    lists:foreach(fun(App) -> application:load(App) end, Apps),
    AppsVsns = lists:map(fun(App) -> {App, get_app_vsn(App)} end, Apps),
    {release, {Name, Version}, {erts, ErtsVsn}, AppsVsns}.

Download the complete script to try it out.

The application I am deploying is a simple echo server, but I won’t go into details here.

You need to package up your application (possibly including the compiled make_release.erl script), then upload it under the ‘Overlay files’ section, and specify a directory to extract the files to. I put the application into /usr/local/, and use the following directory structure:

(Note that the echo-1.0.* files wouldn’t be there when uploaded as they get generated by the script.)

If you’d like the application to be started when the system boots, you may wish to add this to a boot script. Boot scripts can be configured under the ‘Configuration’ section. The following boot script will generate the release file when the system is booted for the first time. Following this, and on subsequent boots, the the Erlang application will be started.

My boot script looks like this:

if [ -f /etc/init.d/suse_studio_firstboot ]
then
  # Generate release for Echo Server
  echo "Generating release for Echo Server..."
  erl -noshell -s make_release -pa /usr/local/Echo-1.0/ebin/
fi
 
# Start the echo server
echo "Starting Echo Server..."
erl -boot echo-1.0 -pa /usr/local/Echo-1.0 -detached

If you’re wondering, it looks like this script gets installed to: /etc/init.d/suse_studio_custom

I installed the ‘open-vm-tools’ package, as suggested (since I had specified that I wanted to generate a VMware virtual machine), and then set the machine to be built.

The appliance took a few minutes to build.

(Having requested a few more builds, it turns out that there’s a queueing system, so you may get lucky, or you may be in for a wait…)

Once the build was complete I was able to try it out in the browser:

After downloading it, double-clicking the .vmx file booted the machine in VMware:

I was then able to test out the Erlang application by connecting with telnet:

Alternative Approaches

An alternative to actually installing your application is to just setup an appliance that has Erlang installed. This would still ease the setup process of the system and would just require you to add your Erlang application in after installation.

If you wanted the latest version of the Erlang system, it might be possible to upload the latest Erlang release as an ‘overlay file’, as opposed to fetching it from the repository. The benefit of using the repository is that it could be set to update automatically.

PHP Script to Compare MySQL Database Schemas

Joe — Mon, 27 Jul 2009 19:40:46 +0000

Introduction

I’ve been working on a project recently that has a rather messy deployment strategy. This resulted in a minor glitch occurring during a recent deployment to a fairly high-profile site when a database table wasn’t updated.

Clearly the best solution to this problem would be to come up with a better deployment strategy, which is something we have been looking into. However, the problem has inspired me to write a small tool which collects information on the schema of a database so that it can be compared with other databases.

A slight twist to the problem comes from the requirement that the databases are more than likely to be hosted on different servers. Since the database servers are setup to only allow connections locally, we aren’t able to access both databases at the same time. Additionally, some of these servers are running PHP 4, so are lacking some newer PHP features.

Fetching Schema Information from MySQL

The MySQL queries to get information about tables are fairly straight-forward.

To get a list of the tables in a database, we can do:

SHOW TABLES

To get information about the fields in a specific table, we can do:

SHOW COLUMNS FROM table_name

This returns a collection of information on each field:

the names of fields,
the data type,
whether null values can be stored,
whether the column is indexed,
the default value of the field, and
whether the field uses auto_increment

Full details can be found on the manual page.

The PHP Script

The tool consists of:

a simple class for extracting the schema and performing the comparison between extracted schemas
a user interface to the class

Since we need to be able to access databases on different hosts, the solution I am using is to serialise the schema information and give it back to the user. The user is then required to copy and paste this data from separate instances back into a form so that the comparison can be performed.

The user interface provides the user with a list of pre-configured databases to choose from, and also allows the user to enter connection details in the form. Arguably, the first option is more convenient and more secure.

The actual comparison of schemas works on three levels:

We check both databases have the same tables
For each table, we check that the same fields are present
For each field, we check that the field’s parameters are the same

Any discrepancies are then displayed to the user on a per-table basis.

Download this Tool

If you have a use for this tool, or are just interested to see how it works, feel free to download the code.

Estimating Pi with Monte Carlo Methods

Joe — Thu, 16 Jul 2009 12:58:16 +0000

This article demonstrates how to use a Monte Carlo simulation to calculate a value for Pi. An example program is given, written in Erlang, which can be run in a parallel environment. As well as the theory behind the method, we will look at how random numbers can be generated in Erlang and how to time Erlang programs. We will also perform an informal analysis of the results.

Monte Carlo Simulations

A Monte Carlo method is the general name of an algorithm that uses a stochastic process to perform repeated random sampling on a problem. A statistical approach is used such that the numerical result to the problem can then be given by considering the number of ‘hits’ that are achieved with the sampling.

The accuracy of the result, and the confidence in it, will increase as the number of samples we take increases.

More details about Monte Carlo methods can be found on the Wikipedia article and in the article The Beginning of the Monte Carlo Method.

Estimating Pi

A nice example of the use of Monte Carlo methods involves estimating a value for Pi. Let us consider a circle (with a radius of one) which is within a square (with edges of length two). Refer to the following diagram:

Now, by choosing random points within the square, we are able to calculate whether or not each point is within the circle or not. We can consider the analogy of throwing darts ‘randomly’ at a dart board.

Since we know the area of a circle with radius r is given by:

And the area of a square with sides of length x is given by:

And in our example, r=1 and x=2 and we know that the ratio, R, of the area of the circle to the total area of the square is:

Therefore, if we can come up with this ratio by experimentation, we can use it to derive a value for Pi:

Generating Random Numbers in Erlang

Our method requires that we generate pairs of random numbers to identify the x- and y-co-ordinates of random points within the square. We need to make sure that these numbers have a uniform distribution—so that there is an equal chance of hitting any point on the dart board.

Fortunately, Erlang has just such a function: random:uniform/0. It is essentially as simple as calling random:uniform() to generate a random float between 0.0 and 1.0. Details can be found in the manual page.

Seeding

A note at the bottom of the manual page states:

If a process calls uniform/0 or uniform/1 without setting a seed first, seed/0 is called automatically.

Interestingly, if a number of processes are spawned together, they will all produce the same list of ‘random’ numbers. This is because each process is seeding the random number generator with the same default values.

To avoid this, we can seed the random number generator ourselves. The manual recommends using the built-in-function now/0. We can achieve this by calling the following line from the newly spawned process:

apply(random, seed, tuple_to_list(now()))

The function now/0 will always return a unique value, so we can be confident that each process has a uniquely seeded random number generator. Incidentally, I am led to believe that now/0 will actually move ahead of the system time if called frequently enough.

Code Listing

-module(pi).
 
-export([start/2]).
 
%% Start processing on the specified number of processes for the specified
%% number of points. The number of points will be split equally amongst the
%% number of processes.
 
start(Processes, Points) ->
	PointsPerProcess = Points div Processes,
	start_processes(Processes, PointsPerProcess),
	wait_for_results(Processes, Points).
 
 
%% Spawn the required number of processes. Each process is passed the Pid of
%% the current process so that it can send the result back.
 
start_processes(0, _PointsPerProcess) ->
	io:format("Started all processes.~n");
 
start_processes(Processes, PointsPerProcess) ->
	Pid = self(),
	spawn(fun() -> process_points(PointsPerProcess, Pid) end),
	start_processes(Processes - 1, PointsPerProcess).
 
 
%% Wait for the results to come back from each of the processeses. The result
%% from each process comes back containing the count of 'hits'. A total sum
%% of hits is calculated, and then when all the results have come back, we
%% give a calculated value for Pi.
%% 
%% Note: this is heavily reliant on the spawned processes not crashing, and
%% returning their respective counts. (Otherwise we will be waiting forever).
 
wait_for_results(Processes, Points) ->
	wait_for_results(Processes, Points, 0).
 
wait_for_results(0, Points, Sum) ->
	Pi = (Sum / Points) * 4,
	io:format("Pi: ~w~n", [Pi]);
 
wait_for_results(Processes, Points, Sum) ->
	receive
		{result, Count} ->
			wait_for_results(Processes - 1, Points, Sum + Count)
	end.
 
 
%% Setup processing of the points. We seed the random number generator, count
%% the number of 'hits' in the simulation, and then send the result back to
%% the parent process.
 
process_points(Points, Pid) ->
	apply(random, seed, tuple_to_list(now())),
	Count = test_points(Points, 0),
	Pid ! {result, Count}.
 
 
%% Each point is chosen randomly and then tested to see whether it is within
%% the circle. If it is, it is added to the count. Finally, the count is
%% returned.
 
test_points(0, Count) -> Count;
 
test_points(Points, Count) ->
	X = (random:uniform() * 2) - 1,
	Y = (random:uniform() * 2) - 1,
	I = X*X + Y*Y,
	if
		I < 1 ->
			test_points(Points - 1, Count + 1);
		true ->
			test_points(Points - 1, Count)
	end.

Note: I’ve only just started learning Erlang. Any feedback on my (simple) code will be welcomed.

Timing an Erlang Program

In order to experiment with running the program on multiple processors, it would be useful to be able to time the execution of the program. There is a module in Erlang for just this purpose. The function to use is: timer:tc/3. The three parameters that it takes are the module, the function and a list of arguments (these three parameters are commonly referred to as the MFA—module, function, arguments).

The following two lines allow us to time the program for ten thousand points running on one process and two processes, respectively:

timer:tc(pi, start, [1, 10000]).
timer:tc(pi, start, [2, 10000]).

The timer:tc/3 function returns the tuple {Time, Value}, where Time is the time that the function took to execute, in microseconds, and Value is the return value from the function we are timing.

Analysis of Results

I’m currently running this code on my MacBook, so as a result of having two CPU cores, running with any more than two processes doesn’t give any speed benefits. It is nice to see, however, that running with two processes does give a significant speed increase over just a single process:

	One Processor	Two Processors	Speed-up
1 000 000 points	1.371	0.6980	1.964
10 000 000 points	13.44	6.854	1.961

Times above (and below) are averaged from three runs of each scenario and given in seconds.

Let’s now look at the estimations given for Pi:

Number of Points	Time taken (seconds)	Estimation for Pi
1 000 000	0.69800	3.14225733
10 000 000	6.8543	3.1418392
100 000 000	68.452	3.14162744
1 000 000 000	843.74	3.14161595

We can observe the convergence to the true value of Pi on a graph:

The blue line shows our estimation of Pi, and the red line shows the true value of Pi. Note that the x-axis has a logarithmic scale.

Extensions

A nice extension to my program would be to be able to leave it running, and be able to call a function to see what the current progress is. For example, the number of points tested, and the current estimation of Pi would be given.

It would also be interesting to see how the program behaved on more than two processors. Specifically, when the processors are distributed over a network. I don’t currently have any other machines set up to try this out on.

Stock Quotes with Ext JS Charts

Joe — Mon, 06 Jul 2009 23:02:01 +0000

I’ll keep the introduction to Ext JS brief, and then give a short primer to using Ext. Skip to the charts section if you’re already familiar with Ext. Or skip straight to the end if you just want to see the final demo.

Ext JS is yet another JavaScript library. However the emphasis is on UI widgets that allow you to build user interfaces in the browser. The concept is far from new, but the way Ext JS stands out from the contenders must be embarrassing. The library’s widgets are visually stunning, feature-rich, and yet still maintains high performance.

I will admit that I hadn’t heard of Ext JS before working on a project last year that used it. I’ve been doing freelance web development for about five years and I tend to keep quite up-to-date with web technologies, so I’m surprised that it had slipped under my radar.

Fairly recently, the team announced the latest release candidate of Ext 3.0. I wanted to experiment with some of the lesser-hyped new features, and so I’ve been working on a small experiment to display stock indexes using the new charts feature.

Hello, world!

Let’s start by walking through a really simple Ext example before we look at the charts.

Download the latest Ext release. Setup a simple HTML page and add a few includes:

1
2
3

At this point, we should be able to load the blank page in the browser without any errors.

The next step is to write some code to do something with the Ext library. So as not to upset the gods, we’re going to make a pop-up ‘hello world’ window. The following is JavaScript code, so put it in a

Joe Freeman's Weblog

Handling Multipart Uploads with MochiWeb

Setting up MochiWeb

Handling File Uploads

Routing the Request

Handling the POSTed multipart data

A Semi-practical Example

Book Review: Ext JS 3.0 Cookbook

The Book’s Structure

Code Samples

Screenshots

Further Observations

Conclusion

A Java Swing Application for Testing PHP Snippets

Overview of the Application

Working with the Command Line from Java

Building the Swing User Interface

JFrame

BorderLayout

JSplitPane

JTextArea and JScrollPane

JButton and Actions

Putting Together the PhpRunner Class

Threads and Thread Safety

Adding a Built-in Web Server

Saving User Settings with the Preferences API

Conclusions

Download

Requirements

FULLTEXT Search with MySQL and CodeIgniter

Getting CodeIgniter Ready

Setting Up the Database

Some Test Data

Searching the Data

CodeIgniter and the MVC Pattern

Adding More Features

Evaluation

Download

Other Search Solutions

Lineographic Interpretations of Images, with an Etch-a-sketch

Selecting the Project

Overview

Feature Extraction

Path Finding

Drawing Techniques

Hystereis

Nxt-A-Sketch

Conclusion

Further Information

Deploying an Erlang Appliance with SUSE Studio

An Introduction to SUSE Studio

My Experiences with SUSE Studio

Setting up an Appliance with Erlang

Alternative Approaches

PHP Script to Compare MySQL Database Schemas

Introduction

Fetching Schema Information from MySQL

The PHP Script

Download this Tool

Estimating Pi with Monte Carlo Methods

Monte Carlo Simulations

Estimating Pi

Generating Random Numbers in Erlang

Seeding

Code Listing

Timing an Erlang Program

Analysis of Results

Extensions

Stock Quotes with Ext JS Charts

Hello, world!

`JFrame`

`BorderLayout`

`JSplitPane`

`JTextArea` and `JScrollPane`

`JButton` and `Action`s

Putting Together the `PhpRunner` Class