Opening a “one box wonder” Xapian search in Python is done via:

xapian.WritableDatabase(db_path, xapian.DB_CREATE_OR_OPEN)

where db_path is the database name that you want to give the index and open the index using:

xapian.Database(db_path)

Since the project uses Pylons, the controller used a path out to the .ini file loaded at runtime to link to the correct database.

Using the documentation on the Xapian site for remote backends and the Python bindings, I was able to quickly adjust the code so that xapian.WritableDatabase is replaced by:

xapian.remote_open_writable(“<host name>”, “<port number>”)

and is opened by:

xapian.remote_open(“<host name>”, “<port number>”)

Once that is set up, then all you need to do is to start the the TCP server which is what I’ve been looking at. I downloaded the tar.gz file of Xapian-core from the Xapian site, configured and made on Ubuntu Lucid Lynx and then ran xapian-tcpsrv –port <port number> <database name> in a new terminal window which allowed me to test the connections and get them ready for going live.

Changes are afoot on the Open Correspondence site as well. As part of a conversation that involved Keith Alexander, of Talis, the project is going to evolve into a slightly more Linked Data direction with references to the books, magazines, correspondents and so on. I’d already started going in this direction with the correspondent links (such as http://www.opencorrespondence.org/letters/correspondent/Miss%20Hogarth) so this is really an extension of where we need to go to connect to other resources such  as Dbpedia, Wikipedia and so on. The fact that it is Dickens’s bi-centenary in 2012 gives an added boost to the project. The Linked Data approach gives us the chance of creating some sort of framework for future expansion and linking together of data sources, not only at a literary level but also socially. It also encourages me to sort out the content negotiation work that was started and to try and follow the FAQs that the Pedantic Web group have posted to make sure that the site follows the best standards that it can and to build them into future developments and directions.

UPDATE: I’ve created an endpoint using the default ARC settings here: http://austgate.co.uk/dickens/endpoint.php

I’ve also been developing a small text ontology to add to the Friend of a Friend (FOAF)  and Dublin Core (DC) ontologies. I’ll post details later. The database schema is still under development but I hope to get that change done soon so that I can get on with the XML changes.

Its originally an extension to the D-Space repository tool but Rufus Pollock used in the Open Knowledge Foundation’s Weaving History project – to which I contributed the Milton json data file. Originally I’d used it just for biographical timelines but thinking about it, I wondered how you could use it to mine datasets like the letters of Charles Dickens.

Dickens was a prolific letter writer (the Pilgrim edition extends to 12 thick volumes). I don’t have access to that data but I did download the first volume (of three) that his daughters edited.

Using Perl, I have extracted the date and recipient tags and converted the text file into JSON (as part of a larger process of converting the file into XML and using XSL to transform the data) and then created a table view of the data so that you can easily find the dates of the letters sent to certain people in tabular form.

I’ve also used the same data set to produce a fairly basic timeline of the letters which is being rewritten from here. It needs some rewriting to update to the new version of timeline.