Wordcloud of Defoe's journey taken at Textcamp

Using the Many Eyes Word Cloud, we can see that Scotland is unsurprisingly the largest item but also Lord and Earl are also popular, suggesting that he stopped with or met the aristocracy rather than just travelling randomly. Dave Flanders and John created some cool visualisations using the tool which allow you to follow words in the text and to see which are the most linked to words (using bigrams I would suppose) in a tree fashion. It is certainly something at I will be looking up later for “quick win” visualisations.

One of the intriguing projects that was suggested was building our own DIY bookscanner using links currently stored on the Textcamp 2011 wiki pages. I think that Dave Flanders might be organising a hack weekend to actually build the machine for real use. I find it interesting but thinking that it would be cool to also see if can be built at home or using iPhone / Android OSes which also entails a software hack, unless an app already exists. That is something to explore later.

Mark MacGillivray, of OKFN and Cottage Labs,  and Brian Hole of Ubiquity Press, spoke about Open Access and making scholarship open but also retaining its rigour. Using Open Access, we should be able to share the data, the ways of interpreting it and and the final interpretation which is published.

The science community has been doing this for some while and things like the Panton Principles and Science Commons are showing the way. One of the ideas was to write a handbook for how to use openness in literature and that it is something that we need address and build on. We ought to write an open guide / manual and build on / develop the Panton Principles where necessary as a core set of principles to work with.

Having days like Textcamp and Book Hackday are extremely useful to think about this and to work on the ideas. It is easy to get into echo chambers of mailing lists and blogs, we need these events to meet new people, be challenged to explain ourselves and to either build on the day or go away with ideas to test and try out. The day has excited me out using word clouds again and doing a bit more work on them as a tool to make them useful. It has also got me excited about book scanning and doing some hardware hacking (which I’ve not really done) before.

Running the Pope essay through Wordle makes me excited about testing what we can do with the ECCO TEI documents that John Levine  links to. Can we hyperlnk to other texts, author and events that are mentioned in it (not just with the annotator tool but in generated HTML) or use HTML 5 to embed audio links to further discussions or pronunciation (for example Byron’s Don Juan which has been argued as pronounced “Jew-an” rather that “Hwan” and the arguments for and against).

Perhaps that gets to one of the issues that arose in the break-out discussions in the kitchen. After the lightning talk about digital publishing, there seemed to be an argument about whether current digital publishing was really pushing the boundaries or flailing around. I do think that it has some real benefits for niche publishing but these have not been fully explored. The model will need to change and perhaps become more open in those senses, perhaps linking the raw data to the interpretation earlier to allow the relevant community to peer review the data earlier. Just a suggestion. There are two distinct communities, the top-down business layer and the grass roots layer, activists, data developers and so on. Both would appear to have broadly similar aims but how to put them together  in a useful way for both to learn. Don’t get me wrong here as I believe I’m at the grass roots layer, but I think that both sides do have a dialogue which could get around the issues that the music and film industries have found themselves in, i.e. confrontation. We are here to disrupt and make.because we are passionate.  We care about the industry. Publishing is an industry which needs to change and transform itself. Put the two together and there are ways of moving forward. My hope is that in future events, we could get some more publishers along to the event.

The other important thing is that these conversations carry on afterwards. The round table discussions where great as were the break-out in the kitchen ones but they need to carry on or we create our own echo chamber which reduces the value of what happened yesterday.

Whilst I did not do as much coding as I wanted to yesterday, I met some new people and caught up with colleagues. The fact that organisations such as JISC are supporting events like this shows their underlying importance and use to the community. We’ve started, now we need to carry on by chatting, blogging, sharing and doing more of these events.

After Book Hackday, I’ve written a post about beginning to develop the Open Correspondence toolkit for the Open Knowledge Foundation’s Notebook blog. I was also contacted regarding converting the TEI XML pages into JSON, which I am currently working on.

Once I’ve done some more work on it, I’ll post the code and more about it.

I’ve been working on another project which may or may not be open. It is certainly interesting but I am not sure I can say much more than that. I hope to have a blog post up soon about it but I am rather excited by it and its possibilities.

Meanwhile, the work project continues apace with some surprising outcomes for me. Following watching a video on Facebook’s architecture, I’m beginning to see certain parts very differently. I really do hope more on this but I’ve got some building to do and a bit more delving and reading that needs completion.

As part of the XML, I’ve been using the TEI mark-up for the letters. I once hard this described as “XML for people who don’t think XML is flexible enough”. Now I can see why. It is a highly flexible solution to digitising texts but can be confusing, especially when switching between versions. I believe the original model that I had been working on was P4 but the current one is P5 so I had to negotiate that change and to make sure that I had the correct elements in the blocks. Even then, there can be two or three different versions of the same element in the section and I do have to wonder about that wisdom rather than simplifying the elements so that there are the extensible elements that may or may not be used. I’m intending to use the schema again and to really get my head around it rather than tinkering on the edges.

I’ve attempted this conversion before but think that I’ve finally got it to a point which is nearly there. What I would really like to do is to put together some sort of tool kit as a core to the Open Correspondence project. Clearly this would be a long-term project and would need more research but it might be useful to other projects.

As well as marking up texts, it would be useful to use the XML mark-up to convert the text into other formats such as Mobipocket or the Kindle formats to allow a user to create their own e-publication. It would also be useful to find a way of using the XML in conjunction with the psbook command to create a print version of a letter or collection. This does mean that I need to convert the XML into a PostScript file (which raises a host of questions at the moment – such as converting structured format into layout format) and then print it.

I’ve also been playing around with the correspondent collections and the way of marking up collections in TEI. I had thought of this as working on creating printable collections and making the data re-usable for printing. Equally it might allow the data to be used in answer to Jonathan Gray’s question regarding identifying the letters written to a particular correspondent.

When I can get the XML working and validated, then I’ll look at the JSON output. It would draw a line under this part of the project and allow me to move on. I’m aiming for a release towards the end of March or middle of April in keeping with trying to keep into a six week schedule.

The next thing after that is to begin answering Jonathan’s questions in terms of a tool kit to identify weaknesses and to try and write some code to re-use and re-mix the data. I would hope that would be in the next release towards the end of May.

One of the driving ideas behind the Open Correspondence RDF was to begin identifying the people to whom Dickens wrote about books. Out of this I would like to create some visualisations of the data. You could possibly do this for the places, for example track his letters for one of the US tours.

But back to the original question. I believe this can be done (as I’ve been working on the XML issues) using Python’s rdflib. The major issue would be to get this working across version 2.4 and 3 so that any released code would be cross-platform.

Jonathan: as an open call, I’d love to work with you on this.

Xapian does word stemming but not lemmas which are slightly different. In stemming, the word production should appear as produc* since produc is the base of the word. However that is nonsense. The base of the word is produce which is what the Wordnet Lemma returns.

Using the API notes, I’ve been playing around with the following:

from nltk.corpus import wordnet as wn

word_lem = set()
ret_lem = []
for i in wn.synsets(author):
[word_lem.add(lemma.name) for lemma in i.lemmas]

ret_lem = list(word_lem)

Having used  set to remove any duplicates, I can return the list of the lemmas that WordNet gives. Since you have to use a Synset if you don’t have the exact part of speech that a word is (Verb, Adverb, Adjective or Noun) since the lemma constructor requires that to produce the lemma. That’s fine  and I can return the names using lemma.name but the part of speech is in the synset and I’m not sure how to retrieve it but it would be useful to send back so that a user can see the part of speech and determine whether it is of interest or not.

In the first instance though, I can return the related synsets to the user through an API, yet to be written, and link them to the Xapian search so that they can search for the term if of interest. It begins the opening up of the letters as a linguistic dataset since the tone and language might vary across the letters depending on the correspondent. One would expect letters to his family to be less formal than to a business colleague or fellow author. I’m aiming to have an early draft up shortly with some improved XML and JSON handling for the individual letters.

Given that I really did not do that well in the linguistics module at the University of Leicester, I’m surprised that this has been the first API module being developed. It makes sense though but I need to find a way of getting back to the original purpose of the site.

The current changes are:

• additional fields in the RDF endpoint.  I still need to do some major work to JSON and XML which I hope to do for the next update.

• a basic text search

• a basic set of geographic data in the collection

• better linking from the letters to the correspondent and geographical data (NB it is still incomplete)

• some mapping with Open Layers javascript.

• a Simile timeline (which is a bit slow at the moment).

Admittedly some of this is exposing work already there but hidden. However I’ve also been working on some unicode fixes to the underlying XML which is used by the project which has meant rebuilding the tables and the Xapian indexes.

Following a request on the Open Literature mailing list, I’m looking at the idea of using Python’s NLTK to create some linguistic API wrappers around the Xapian search. It strikes me that these letters can be used to create a corpus of Dickens’s language where you can explore the language used in family correspondence (to his daughters and wife), to other authors (Wilkie Collins) and to readers. That is a longer project though in terms of building the relevant indexes.

I’m also looking at the idea of clustering a collection of letters to a correspondent and seeing what happens (for some reason, the current script is looking at Wilkie Collins). There is also a set of queries that one might run against letters discusing books and the publication dates to view the distribution. I’m working on these latter questions at the moment for intended release later this week but I do foresee it being delayed a while.

Firstly, I need to work out a more exact way of mapping the data in the database or flat file. I think what I really need is to use something like:

• place
• address
• city
• latitude
• longitude
• description
• url

The data that I have is not quite as granular as this. Yet. When I’ve done this, I need to build the mapping so that if a place is entered, say Hotel Meurice, Paris, then I can return the details and latitude / longitude to render an Open layers map. That’s almost the easiest bit really.

The second issue is the difference in names. Over time and in the heat of writing, names can change subtly. For instance Gads Hill Place, one of Dickens’s homes which is now a school. In the letters it is referred to as

1. Gad’s Hill Place,
2. Gad’s Hill Place, Higham
3. Gad’s Hill

It can also be known as Gadshill Place or Gads Hill Place. I need to find a way of differencing the terms. Firstly I need to develop a way of checking inside a term and then returning it if it is a new terms or returning the mapped version if it matches a term. Secondly I need to fuzzy match the strings so that any near differences (using the Levenshtein edit distance) can be checked and either ignored or exclude the term.

These issues will also affect the correspondent code which is being created. I suspect that anything with names will have the same issues. For  instance, Wilkie Collins is known in the letters as Mr W Wilkie Collins and Mr Wilkie Collins. In the current implementation of the site, these are two different entities which is clearly wrong. They are the same entity but there is a subtle difference which is not accounted.

So to deal with this, I am going back to the parsing library and building these in instead. Whilst it is a slower way of dealing with these issues, it provides a chance of doing any necessary information and site re-thinking.

As part of this, I downloaded some TEI guidelines from the California Digital Library to use to build the base metadata export. Ideally what I’m hoping to do is to create the data as a Python dictionary and then reformat into HTML, HTML & RDFa, RDF, JSON or XML. It should allow me to export the same data for each type.

I’m sure at times I’ll wonder what I started but it needs doing if the site is to accept more authors. After that, back to search.

On a separate note, I have also done some work on the Arts Funding search. I’ve given it a re-skin and used the Accordion widget from the JQuery UI. It also has some more search options built in so that the data can be searched by date and amount as well as political constituency and art form. The search needs to take in some arguments such as < or > or equals in the amount but that can come. I’ve been reading Jenni Tennison’s post on the data.gov.uk blog to best expose the data using Linked Data.

Whilst writing this post, it occurs to me that whilst Linked Data is an awesome way of exposing data, useful search is still an important part of any content driven website. As blogged before, I have implemented an early version of a Xapian search. As Tim Bray has noted, advanced search might have a smaller use but it is more likely to be used by the heavier users so deserves to have time taken on it.

As part of the search, I came across a reference to Sean Gillies’ useful blog post talking about modelling historical place names for the Pleiades project. What intrigues me about the places is that they don’t exist in amber. They change and adapt.

Playing around with Open Layers (and inspired by Jo Walsh’s piece on historical maps on Mapping Hacks), I’ve become interested in the idea of placing a historical map on top of a current street map so that you can see what a place looks like now and also when, for example, Dickens lived in Tavistock Square or Gad’s Hill Place. How has it changed? What did it look like then? What does it look like now? Does it even exist?

Whilst that may not aid textual analysis, it could be tied into historical and social queries about the letters. By adding this layer of data, which one might not normally think about in terms of leters data, we can find out other things of interest.

I think for now, I’ll try not to go too far down this road only so that I can get the other bits of data fixed first.

I’ve gone back to the some of the work that I was doing for the first version of the site way back last year. As part of the move to Linked Data, I’ve been working on a URI for places and books. Places, asn oted in previous posts, has come together and is just in need of some tidying up. I’ve managed to create an index page from the RDF endpoint using rdflib to parse the triples looking for the geo: namespace and then putting the items into a set to remove the duplicates. This needs changing as sets are unordered and I’d like the page to be ordered so that a pace can be found quickly. Perhaps a better option would be to place the raw data into a dictionary and cast to a list to sort at the last moment (or more simply sort the keys in the dictionary…) and then to remove the duplicates such as Gad’s Hill which is analogous to Gadshill. Both are used but refer to the same entity, so I need to do a difference pn the string (probably using difflib or a variant)  to identify the changes and clean up the URIs.

With the books, I had created a table of the publication dates and the titles, so all I need to do is to map the book’s variant titles, such as the “The Adventures of Nicholas Nickleby” is better known as “Nicholas Nickelby” or plain “Nickleby” in the letters. It might be easiest to put this into a dictionary at the moment rather than another table and to call that. I would also need to get some sort of introduction (and perhaps in the future create an Open Dickens site for the novels).

I’m sure I can do this in a few hours and to get it working. Must make the time now I’ve had a small break.

Last night I managed to build the places index, the locations from where Charles Dickens wrote his letters, parsing the RDF with rdflib 3 (though I’ve been trying to enure that it is compatible with version 2.4.2 on Linux though that is to come shortly). It still needs tidying up but I’m going to post what I have with the next draft of the site. Dickens’s letters cover France, the US and the UK and gradually I’m getting the latitude and longtitudes for the locations so that they can be used with Open Layers maps.

Does the project necessarily break new ground? Possibly, possibly not. I’m not a Dickens scholar (though I do think he could turn a phrase or two…) I hope that the project will allow us to think about ways of linking and sharing the data as well as find different ways of mining it. Mapping seems like one query. Placing the letters into context as well using historical data would be useful (but I need to find the right data sources – not just Wikipedia).

In all, I firmly believe that it is important to create and experiment with the data available and to think of new queries or mash-ups. What we create now will probably be placed into the shadows by the next generation but why wait. Let’s have fun now.