Open Correspondence

Development on the Open Correspondence project has been slow to stalled for a while. I have been doing bits and pieces but sitting down with Mark McGillivray of Cottage Labs and the Open Knowledge Foundation, brought some clarity. Recently the Textus project has been announced and I have been talking with the developers to put the data onto that platform. It seems to me that it is better to pool resources and to contribute where I can. There are parts of the existing project that I like and others that need more work to make me happy and it seems right now to move onto the developing platform.

Textcamp

At Textcamp last September, one of the sessions covered DIY Bookscanners (Austgate post on Textcamp). One of the actions on the Textus wiki was OCRing text. I have posted previously about playing with Tesseract and seeing this, I emailed the humanities-dev list to explore the possibilities. To this end, I have volunteered to work on the area and will write a blog post about it There is already a large amount of work that exists, so  I am perhaps not developing anything new. However it would, I think, be interesting to develop a stand-alone system that is flexible and downloadable. Like other OKF projects, it will be a Python project but also be a hardware project to try and extend some of the existing projects.

Other Bits

I’ve been working on an indexing project which appears to be coming together quite nicely. Hopefully I’ll be able to say some more shortly but it depends on a conversation that has yet to be had.

Next week, after a break, is a return to work and to data. The Dev8d conference provided me with some ideas and clarity on one or two things, so time to put them into practice.

As part of the XML, I’ve been using the TEI mark-up for the letters. I once hard this described as “XML for people who don’t think XML is flexible enough”. Now I can see why. It is a highly flexible solution to digitising texts but can be confusing, especially when switching between versions. I believe the original model that I had been working on was P4 but the current one is P5 so I had to negotiate that change and to make sure that I had the correct elements in the blocks. Even then, there can be two or three different versions of the same element in the section and I do have to wonder about that wisdom rather than simplifying the elements so that there are the extensible elements that may or may not be used. I’m intending to use the schema again and to really get my head around it rather than tinkering on the edges.

I’ve attempted this conversion before but think that I’ve finally got it to a point which is nearly there. What I would really like to do is to put together some sort of tool kit as a core to the Open Correspondence project. Clearly this would be a long-term project and would need more research but it might be useful to other projects.

As well as marking up texts, it would be useful to use the XML mark-up to convert the text into other formats such as Mobipocket or the Kindle formats to allow a user to create their own e-publication. It would also be useful to find a way of using the XML in conjunction with the psbook command to create a print version of a letter or collection. This does mean that I need to convert the XML into a PostScript file (which raises a host of questions at the moment – such as converting structured format into layout format) and then print it.

I’ve also been playing around with the correspondent collections and the way of marking up collections in TEI. I had thought of this as working on creating printable collections and making the data re-usable for printing. Equally it might allow the data to be used in answer to Jonathan Gray’s question regarding identifying the letters written to a particular correspondent.

When I can get the XML working and validated, then I’ll look at the JSON output. It would draw a line under this part of the project and allow me to move on. I’m aiming for a release towards the end of March or middle of April in keeping with trying to keep into a six week schedule.

The next thing after that is to begin answering Jonathan’s questions in terms of a tool kit to identify weaknesses and to try and write some code to re-use and re-mix the data. I would hope that would be in the next release towards the end of May.

One of the driving ideas behind the Open Correspondence RDF was to begin identifying the people to whom Dickens wrote about books. Out of this I would like to create some visualisations of the data. You could possibly do this for the places, for example track his letters for one of the US tours.

But back to the original question. I believe this can be done (as I’ve been working on the XML issues) using Python’s rdflib. The major issue would be to get this working across version 2.4 and 3 so that any released code would be cross-platform.

Jonathan: as an open call, I’d love to work with you on this.

Xapian does word stemming but not lemmas which are slightly different. In stemming, the word production should appear as produc* since produc is the base of the word. However that is nonsense. The base of the word is produce which is what the Wordnet Lemma returns.

Using the API notes, I’ve been playing around with the following:

from nltk.corpus import wordnet as wn

word_lem = set()
ret_lem = []
for i in wn.synsets(author):
[word_lem.add(lemma.name) for lemma in i.lemmas]

ret_lem = list(word_lem)

Having used  set to remove any duplicates, I can return the list of the lemmas that WordNet gives. Since you have to use a Synset if you don’t have the exact part of speech that a word is (Verb, Adverb, Adjective or Noun) since the lemma constructor requires that to produce the lemma. That’s fine  and I can return the names using lemma.name but the part of speech is in the synset and I’m not sure how to retrieve it but it would be useful to send back so that a user can see the part of speech and determine whether it is of interest or not.

In the first instance though, I can return the related synsets to the user through an API, yet to be written, and link them to the Xapian search so that they can search for the term if of interest. It begins the opening up of the letters as a linguistic dataset since the tone and language might vary across the letters depending on the correspondent. One would expect letters to his family to be less formal than to a business colleague or fellow author. I’m aiming to have an early draft up shortly with some improved XML and JSON handling for the individual letters.

Given that I really did not do that well in the linguistics module at the University of Leicester, I’m surprised that this has been the first API module being developed. It makes sense though but I need to find a way of getting back to the original purpose of the site.

The current changes are:

• additional fields in the RDF endpoint.  I still need to do some major work to JSON and XML which I hope to do for the next update.

• a basic text search

• a basic set of geographic data in the collection

• better linking from the letters to the correspondent and geographical data (NB it is still incomplete)

• some mapping with Open Layers javascript.

• a Simile timeline (which is a bit slow at the moment).

Admittedly some of this is exposing work already there but hidden. However I’ve also been working on some unicode fixes to the underlying XML which is used by the project which has meant rebuilding the tables and the Xapian indexes.

Following a request on the Open Literature mailing list, I’m looking at the idea of using Python’s NLTK to create some linguistic API wrappers around the Xapian search. It strikes me that these letters can be used to create a corpus of Dickens’s language where you can explore the language used in family correspondence (to his daughters and wife), to other authors (Wilkie Collins) and to readers. That is a longer project though in terms of building the relevant indexes.

I’m also looking at the idea of clustering a collection of letters to a correspondent and seeing what happens (for some reason, the current script is looking at Wilkie Collins). There is also a set of queries that one might run against letters discusing books and the publication dates to view the distribution. I’m working on these latter questions at the moment for intended release later this week but I do foresee it being delayed a while.

I’ve finally done the first pass of this update which I’m hoping to make live soon once I’ve updated the controllers and re-checked the other data improvements. Whilst it is not perfect, it is a lot better than it was. I think that the next week will be spent going over the endpoints and the Pylons controllers so that the data is cleaner than at present and correctly linked.

It has been a useful exercise in that I’ve started rewriting the parser for the letters (an ongoing large job I was thinking of doing when I come to the next set of letters) and putting some of the earlier thoughts into place.

Once I’m happy with these updates, I’ll update the site which does mean rebuilding the databases and endpoints. However once it is done, it should be a lot cleaner  and I can then start looking at the correspondents and linking into other data sources like dbpedia.org. I think that the first task though might be to restart work on the clients that I had been putting together  as a basic development kit.

Firstly, I need to work out a more exact way of mapping the data in the database or flat file. I think what I really need is to use something like:

• place
• address
• city
• latitude
• longitude
• description
• url

The data that I have is not quite as granular as this. Yet. When I’ve done this, I need to build the mapping so that if a place is entered, say Hotel Meurice, Paris, then I can return the details and latitude / longitude to render an Open layers map. That’s almost the easiest bit really.

The second issue is the difference in names. Over time and in the heat of writing, names can change subtly. For instance Gads Hill Place, one of Dickens’s homes which is now a school. In the letters it is referred to as

1. Gad’s Hill Place,
2. Gad’s Hill Place, Higham
3. Gad’s Hill

It can also be known as Gadshill Place or Gads Hill Place. I need to find a way of differencing the terms. Firstly I need to develop a way of checking inside a term and then returning it if it is a new terms or returning the mapped version if it matches a term. Secondly I need to fuzzy match the strings so that any near differences (using the Levenshtein edit distance) can be checked and either ignored or exclude the term.

These issues will also affect the correspondent code which is being created. I suspect that anything with names will have the same issues. For  instance, Wilkie Collins is known in the letters as Mr W Wilkie Collins and Mr Wilkie Collins. In the current implementation of the site, these are two different entities which is clearly wrong. They are the same entity but there is a subtle difference which is not accounted.

So to deal with this, I am going back to the parsing library and building these in instead. Whilst it is a slower way of dealing with these issues, it provides a chance of doing any necessary information and site re-thinking.

As part of this, I downloaded some TEI guidelines from the California Digital Library to use to build the base metadata export. Ideally what I’m hoping to do is to create the data as a Python dictionary and then reformat into HTML, HTML & RDFa, RDF, JSON or XML. It should allow me to export the same data for each type.

I’m sure at times I’ll wonder what I started but it needs doing if the site is to accept more authors. After that, back to search.

On a separate note, I have also done some work on the Arts Funding search. I’ve given it a re-skin and used the Accordion widget from the JQuery UI. It also has some more search options built in so that the data can be searched by date and amount as well as political constituency and art form. The search needs to take in some arguments such as < or > or equals in the amount but that can come. I’ve been reading Jenni Tennison’s post on the data.gov.uk blog to best expose the data using Linked Data.

Whilst writing this post, it occurs to me that whilst Linked Data is an awesome way of exposing data, useful search is still an important part of any content driven website. As blogged before, I have implemented an early version of a Xapian search. As Tim Bray has noted, advanced search might have a smaller use but it is more likely to be used by the heavier users so deserves to have time taken on it.

As part of the search, I came across a reference to Sean Gillies’ useful blog post talking about modelling historical place names for the Pleiades project. What intrigues me about the places is that they don’t exist in amber. They change and adapt.

Playing around with Open Layers (and inspired by Jo Walsh’s piece on historical maps on Mapping Hacks), I’ve become interested in the idea of placing a historical map on top of a current street map so that you can see what a place looks like now and also when, for example, Dickens lived in Tavistock Square or Gad’s Hill Place. How has it changed? What did it look like then? What does it look like now? Does it even exist?

Whilst that may not aid textual analysis, it could be tied into historical and social queries about the letters. By adding this layer of data, which one might not normally think about in terms of leters data, we can find out other things of interest.

I think for now, I’ll try not to go too far down this road only so that I can get the other bits of data fixed first.

I’ve gone back to the some of the work that I was doing for the first version of the site way back last year. As part of the move to Linked Data, I’ve been working on a URI for places and books. Places, asn oted in previous posts, has come together and is just in need of some tidying up. I’ve managed to create an index page from the RDF endpoint using rdflib to parse the triples looking for the geo: namespace and then putting the items into a set to remove the duplicates. This needs changing as sets are unordered and I’d like the page to be ordered so that a pace can be found quickly. Perhaps a better option would be to place the raw data into a dictionary and cast to a list to sort at the last moment (or more simply sort the keys in the dictionary…) and then to remove the duplicates such as Gad’s Hill which is analogous to Gadshill. Both are used but refer to the same entity, so I need to do a difference pn the string (probably using difflib or a variant)  to identify the changes and clean up the URIs.

With the books, I had created a table of the publication dates and the titles, so all I need to do is to map the book’s variant titles, such as the “The Adventures of Nicholas Nickleby” is better known as “Nicholas Nickelby” or plain “Nickleby” in the letters. It might be easiest to put this into a dictionary at the moment rather than another table and to call that. I would also need to get some sort of introduction (and perhaps in the future create an Open Dickens site for the novels).

I’m sure I can do this in a few hours and to get it working. Must make the time now I’ve had a small break.

Opening a “one box wonder” Xapian search in Python is done via:

xapian.WritableDatabase(db_path, xapian.DB_CREATE_OR_OPEN)

where db_path is the database name that you want to give the index and open the index using:

xapian.Database(db_path)

Since the project uses Pylons, the controller used a path out to the .ini file loaded at runtime to link to the correct database.

Using the documentation on the Xapian site for remote backends and the Python bindings, I was able to quickly adjust the code so that xapian.WritableDatabase is replaced by:

xapian.remote_open_writable(“<host name>”, “<port number>”)

and is opened by:

xapian.remote_open(“<host name>”, “<port number>”)

Once that is set up, then all you need to do is to start the the TCP server which is what I’ve been looking at. I downloaded the tar.gz file of Xapian-core from the Xapian site, configured and made on Ubuntu Lucid Lynx and then ran xapian-tcpsrv –port <port number> <database name> in a new terminal window which allowed me to test the connections and get them ready for going live.

Changes are afoot on the Open Correspondence site as well. As part of a conversation that involved Keith Alexander, of Talis, the project is going to evolve into a slightly more Linked Data direction with references to the books, magazines, correspondents and so on. I’d already started going in this direction with the correspondent links (such as http://www.opencorrespondence.org/letters/correspondent/Miss%20Hogarth) so this is really an extension of where we need to go to connect to other resources such  as Dbpedia, Wikipedia and so on. The fact that it is Dickens’s bi-centenary in 2012 gives an added boost to the project. The Linked Data approach gives us the chance of creating some sort of framework for future expansion and linking together of data sources, not only at a literary level but also socially. It also encourages me to sort out the content negotiation work that was started and to try and follow the FAQs that the Pedantic Web group have posted to make sure that the site follows the best standards that it can and to build them into future developments and directions.