Using Python’s NLTK library, I managed to extract around 60 distinct locations from his novel, Bleak House. A bit of human editing has tidied this up for me but it looks useful.

Having popped this into MySQL as I am more familiar with it than PostgreSQL and returned it using Open Layers, I’ve been looking at ways of having a query which takes the user location (once given permission) and then calculates locations that around a mile away. Or within walking distance. A bit of searching around led me to this Stackoverflow question on finding distances within a certain radius from a given location. The post is also helpful in pointing me towards some of the maths so that when I have some time, I can try to understand the underlying query.

Anyhow, HTML5 to some degree has come to the rescue with its geolocation API. I know it is not a standard at the moment (the draft I saw was 28th June, 2011), but I thought it would be fun to use it to start doing some mapping with the API, courtesy of some guidance from HTML5Doctor and HTML5demos site. It also helps me to re-use some of the data collected for the Open Correspondence project and to bring locations related to Dickens together and make them useful.

What would be interesting to see is the locations surrounding a given latitude and longitude within one mile so that the user could walk to them or find out about them and query them to see if they really want to go to that location.

Opening a “one box wonder” Xapian search in Python is done via:

xapian.WritableDatabase(db_path, xapian.DB_CREATE_OR_OPEN)

where db_path is the database name that you want to give the index and open the index using:

xapian.Database(db_path)

Since the project uses Pylons, the controller used a path out to the .ini file loaded at runtime to link to the correct database.

Using the documentation on the Xapian site for remote backends and the Python bindings, I was able to quickly adjust the code so that xapian.WritableDatabase is replaced by:

xapian.remote_open_writable(“<host name>”, “<port number>”)

and is opened by:

xapian.remote_open(“<host name>”, “<port number>”)

Once that is set up, then all you need to do is to start the the TCP server which is what I’ve been looking at. I downloaded the tar.gz file of Xapian-core from the Xapian site, configured and made on Ubuntu Lucid Lynx and then ran xapian-tcpsrv –port <port number> <database name> in a new terminal window which allowed me to test the connections and get them ready for going live.

Changes are afoot on the Open Correspondence site as well. As part of a conversation that involved Keith Alexander, of Talis, the project is going to evolve into a slightly more Linked Data direction with references to the books, magazines, correspondents and so on. I’d already started going in this direction with the correspondent links (such as http://www.opencorrespondence.org/letters/correspondent/Miss%20Hogarth) so this is really an extension of where we need to go to connect to other resources such  as Dbpedia, Wikipedia and so on. The fact that it is Dickens’s bi-centenary in 2012 gives an added boost to the project. The Linked Data approach gives us the chance of creating some sort of framework for future expansion and linking together of data sources, not only at a literary level but also socially. It also encourages me to sort out the content negotiation work that was started and to try and follow the FAQs that the Pedantic Web group have posted to make sure that the site follows the best standards that it can and to build them into future developments and directions.

UPDATE: I’ve created an endpoint using the default ARC settings here: http://austgate.co.uk/dickens/endpoint.php

I’ve also been developing a small text ontology to add to the Friend of a Friend (FOAF)  and Dublin Core (DC) ontologies. I’ll post details later. The database schema is still under development but I hope to get that change done soon so that I can get on with the XML changes.

Its originally an extension to the D-Space repository tool but Rufus Pollock used in the Open Knowledge Foundation’s Weaving History project – to which I contributed the Milton json data file. Originally I’d used it just for biographical timelines but thinking about it, I wondered how you could use it to mine datasets like the letters of Charles Dickens.

Dickens was a prolific letter writer (the Pilgrim edition extends to 12 thick volumes). I don’t have access to that data but I did download the first volume (of three) that his daughters edited.

Using Perl, I have extracted the date and recipient tags and converted the text file into JSON (as part of a larger process of converting the file into XML and using XSL to transform the data) and then created a table view of the data so that you can easily find the dates of the letters sent to certain people in tabular form.

I’ve also used the same data set to produce a fairly basic timeline of the letters which is being rewritten from here. It needs some rewriting to update to the new version of timeline.