I decided to install it on Snow Leopard, having already set up PEAR and PECL. Using pecl install apc, I downloaded APC which appeared to be fine. However when I ran some scripts, I got the error:
Fatal error: Unknown: apc_fcntl_unlock failed: in Unknown on line 0

which was definitely not expected.

I found some useful instructions on the PHP Bug fixes (https://bugs.php.net/bug.php?id=59750 ). I ran pecl uninstall apc then installed the beta version and answered “no” to pthreadm utex locks and “yes” to spinlocks. You will also need to add apc.rfc1867 = 1 to the php.ini file for use with Drupal 7.

Having done this, I no longer get any errors in the status report regarding APC and when I have got my head around Organic Groups, I’m looking forward to getting into APC and improving Drupal performance.

One of the matters that did concern us was the word “data” and what this might mean to literature and humanities. One assumption that we had was that it perhaps had a more defined meaning to scientists. But what is data to humanities? Is it the manuscript, the notes, or the published work? We decided that ‘Work’ might be a better word for the overarching principle.

One of the issues that is important is re-use and subsequently closing the re-used work down and making it non-open. The major party that we had in mind was Google Books. Whilst they are making good and admirable strides in the digitising of out of print works but there is no API or metadata store that can be used to mix up the data or to mine it in any other way. Effectively we end up where we started: with a technically open text tied up in ways that cannot be re-used.

Re-use and re-mix are extremely important within digital humanities. Influence and building on works are central to movements like Modernism and also ensuring that works and authors are accessible. Works are adapted and take on their own lives or segue from such moments.

The final major point was that citations and the underlying cited text should be open. Whilst the core of the principles are about the work and ensuring that it can be worked on, a fair amount of work goes into notes and annotations to the text (such as the great Annotate It tool) and these provide a meta work for people to build on. It is vital for debate that these are not put into a closed arena, not just for the sharing of notes but also building on the notes. They might also be put together into a new work or an annotated version of a work put together to build upon the work with communal notes.

This does represent a step forward in open literature and digital humanities. I really hope that debate does start and that these can be developed and make concrete.

I’ve just come across this post from Philippe Agrain  on his blog (originally linked from OKF’s Open Humanities mailing list) regarding the recent agreement signed by the British Library and Google. The upshot appears to be that the present and future rights to public domain work has been given to Google. Sorry, public domain should stay in the public domain.

I was at a conference in Oxford a couple of years ago where some of these issues were being discussed and it sounded like the Library were having to do these sort of deals with Google (and, I believe at the time, Microsoft – who pulled out eventually) through the costs and technical expertise needed. I’m pretty sure that these could be sourced or bodged in Britain and Europe if the will is there. I rather fear, in the UK at least, that we do not have the will here particularly at the moment. Yet again, we might be echoing Eric Schmidt’s point made in the recent MacTaggart speech at the 2011 Edinburgh TV festival: that the UK helped invent so many things and then failed to follow through. Rather than being in the technical sphere, we’re now doing this in the cultural sphere – and this surely cannot be right, can it?

The Open Knowledge Foundation, with Creative Commons and Centrum Cyfrowe, are organising a workshop on opening cultural, library and museum metadata.

One hopes that the British Library are going or any of our cultural mavens. We have a rich cultural heritage, we cannot have it locked away in certain companies.

Wordcloud of Defoe's journey taken at Textcamp

Using the Many Eyes Word Cloud, we can see that Scotland is unsurprisingly the largest item but also Lord and Earl are also popular, suggesting that he stopped with or met the aristocracy rather than just travelling randomly. Dave Flanders and John created some cool visualisations using the tool which allow you to follow words in the text and to see which are the most linked to words (using bigrams I would suppose) in a tree fashion. It is certainly something at I will be looking up later for “quick win” visualisations.

One of the intriguing projects that was suggested was building our own DIY bookscanner using links currently stored on the Textcamp 2011 wiki pages. I think that Dave Flanders might be organising a hack weekend to actually build the machine for real use. I find it interesting but thinking that it would be cool to also see if can be built at home or using iPhone / Android OSes which also entails a software hack, unless an app already exists. That is something to explore later.

Mark MacGillivray, of OKFN and Cottage Labs,  and Brian Hole of Ubiquity Press, spoke about Open Access and making scholarship open but also retaining its rigour. Using Open Access, we should be able to share the data, the ways of interpreting it and and the final interpretation which is published.

The science community has been doing this for some while and things like the Panton Principles and Science Commons are showing the way. One of the ideas was to write a handbook for how to use openness in literature and that it is something that we need address and build on. We ought to write an open guide / manual and build on / develop the Panton Principles where necessary as a core set of principles to work with.

Having days like Textcamp and Book Hackday are extremely useful to think about this and to work on the ideas. It is easy to get into echo chambers of mailing lists and blogs, we need these events to meet new people, be challenged to explain ourselves and to either build on the day or go away with ideas to test and try out. The day has excited me out using word clouds again and doing a bit more work on them as a tool to make them useful. It has also got me excited about book scanning and doing some hardware hacking (which I’ve not really done) before.

Running the Pope essay through Wordle makes me excited about testing what we can do with the ECCO TEI documents that John Levine  links to. Can we hyperlnk to other texts, author and events that are mentioned in it (not just with the annotator tool but in generated HTML) or use HTML 5 to embed audio links to further discussions or pronunciation (for example Byron’s Don Juan which has been argued as pronounced “Jew-an” rather that “Hwan” and the arguments for and against).

Perhaps that gets to one of the issues that arose in the break-out discussions in the kitchen. After the lightning talk about digital publishing, there seemed to be an argument about whether current digital publishing was really pushing the boundaries or flailing around. I do think that it has some real benefits for niche publishing but these have not been fully explored. The model will need to change and perhaps become more open in those senses, perhaps linking the raw data to the interpretation earlier to allow the relevant community to peer review the data earlier. Just a suggestion. There are two distinct communities, the top-down business layer and the grass roots layer, activists, data developers and so on. Both would appear to have broadly similar aims but how to put them together  in a useful way for both to learn. Don’t get me wrong here as I believe I’m at the grass roots layer, but I think that both sides do have a dialogue which could get around the issues that the music and film industries have found themselves in, i.e. confrontation. We are here to disrupt and make.because we are passionate.  We care about the industry. Publishing is an industry which needs to change and transform itself. Put the two together and there are ways of moving forward. My hope is that in future events, we could get some more publishers along to the event.

The other important thing is that these conversations carry on afterwards. The round table discussions where great as were the break-out in the kitchen ones but they need to carry on or we create our own echo chamber which reduces the value of what happened yesterday.

Whilst I did not do as much coding as I wanted to yesterday, I met some new people and caught up with colleagues. The fact that organisations such as JISC are supporting events like this shows their underlying importance and use to the community. We’ve started, now we need to carry on by chatting, blogging, sharing and doing more of these events.

After Book Hackday, I’ve written a post about beginning to develop the Open Correspondence toolkit for the Open Knowledge Foundation’s Notebook blog. I was also contacted regarding converting the TEI XML pages into JSON, which I am currently working on.

Once I’ve done some more work on it, I’ll post the code and more about it.

I’ve been working on another project which may or may not be open. It is certainly interesting but I am not sure I can say much more than that. I hope to have a blog post up soon about it but I am rather excited by it and its possibilities.

Meanwhile, the work project continues apace with some surprising outcomes for me. Following watching a video on Facebook’s architecture, I’m beginning to see certain parts very differently. I really do hope more on this but I’ve got some building to do and a bit more delving and reading that needs completion.

I’ve just spent the evening using Node.js and Redis, making a change from usually using Redis with PHP or occasional Python. I’ve been using the latter for logging work and messaging but have been intrigued by the idea of having a system constantly pushing out annotations if added to a document between distributed editors. The idea is that these would all be saved and pushed out to users of a particular document and stored in Redis.

Node strikes me as a good way of creating an efficient back-end system in polling Redis (or perhaps using Redis’s Pub/Sub architecture to achieve a similar end). I would need to look at whether the Publish command would save the data though.

From this evening’s hack, I’ve got the polling from the server working and need to separate out the pushing into the store from the polling to retrieve all notes/comments/et al. I suppose the next thing after that is to pop in an html page or template for a front end.

I’ll do some more tomorrow but I’m making headway with Node which is cool as I’ve been trying to find something meatier to do with it before introducing it into work, even as a development project.

As part of the XML, I’ve been using the TEI mark-up for the letters. I once hard this described as “XML for people who don’t think XML is flexible enough”. Now I can see why. It is a highly flexible solution to digitising texts but can be confusing, especially when switching between versions. I believe the original model that I had been working on was P4 but the current one is P5 so I had to negotiate that change and to make sure that I had the correct elements in the blocks. Even then, there can be two or three different versions of the same element in the section and I do have to wonder about that wisdom rather than simplifying the elements so that there are the extensible elements that may or may not be used. I’m intending to use the schema again and to really get my head around it rather than tinkering on the edges.

I’ve attempted this conversion before but think that I’ve finally got it to a point which is nearly there. What I would really like to do is to put together some sort of tool kit as a core to the Open Correspondence project. Clearly this would be a long-term project and would need more research but it might be useful to other projects.

As well as marking up texts, it would be useful to use the XML mark-up to convert the text into other formats such as Mobipocket or the Kindle formats to allow a user to create their own e-publication. It would also be useful to find a way of using the XML in conjunction with the psbook command to create a print version of a letter or collection. This does mean that I need to convert the XML into a PostScript file (which raises a host of questions at the moment – such as converting structured format into layout format) and then print it.

I’ve also been playing around with the correspondent collections and the way of marking up collections in TEI. I had thought of this as working on creating printable collections and making the data re-usable for printing. Equally it might allow the data to be used in answer to Jonathan Gray’s question regarding identifying the letters written to a particular correspondent.

When I can get the XML working and validated, then I’ll look at the JSON output. It would draw a line under this part of the project and allow me to move on. I’m aiming for a release towards the end of March or middle of April in keeping with trying to keep into a six week schedule.

The next thing after that is to begin answering Jonathan’s questions in terms of a tool kit to identify weaknesses and to try and write some code to re-use and re-mix the data. I would hope that would be in the next release towards the end of May.

One of the driving ideas behind the Open Correspondence RDF was to begin identifying the people to whom Dickens wrote about books. Out of this I would like to create some visualisations of the data. You could possibly do this for the places, for example track his letters for one of the US tours.

But back to the original question. I believe this can be done (as I’ve been working on the XML issues) using Python’s rdflib. The major issue would be to get this working across version 2.4 and 3 so that any released code would be cross-platform.

Jonathan: as an open call, I’d love to work with you on this.

Xapian does word stemming but not lemmas which are slightly different. In stemming, the word production should appear as produc* since produc is the base of the word. However that is nonsense. The base of the word is produce which is what the Wordnet Lemma returns.

Using the API notes, I’ve been playing around with the following:

from nltk.corpus import wordnet as wn

word_lem = set()
ret_lem = []
for i in wn.synsets(author):
[word_lem.add(lemma.name) for lemma in i.lemmas]

ret_lem = list(word_lem)

Having used  set to remove any duplicates, I can return the list of the lemmas that WordNet gives. Since you have to use a Synset if you don’t have the exact part of speech that a word is (Verb, Adverb, Adjective or Noun) since the lemma constructor requires that to produce the lemma. That’s fine  and I can return the names using lemma.name but the part of speech is in the synset and I’m not sure how to retrieve it but it would be useful to send back so that a user can see the part of speech and determine whether it is of interest or not.

In the first instance though, I can return the related synsets to the user through an API, yet to be written, and link them to the Xapian search so that they can search for the term if of interest. It begins the opening up of the letters as a linguistic dataset since the tone and language might vary across the letters depending on the correspondent. One would expect letters to his family to be less formal than to a business colleague or fellow author. I’m aiming to have an early draft up shortly with some improved XML and JSON handling for the individual letters.

Given that I really did not do that well in the linguistics module at the University of Leicester, I’m surprised that this has been the first API module being developed. It makes sense though but I need to find a way of getting back to the original purpose of the site.

The current changes are:

• additional fields in the RDF endpoint.  I still need to do some major work to JSON and XML which I hope to do for the next update.

• a basic text search

• a basic set of geographic data in the collection

• better linking from the letters to the correspondent and geographical data (NB it is still incomplete)

• some mapping with Open Layers javascript.

• a Simile timeline (which is a bit slow at the moment).

Admittedly some of this is exposing work already there but hidden. However I’ve also been working on some unicode fixes to the underlying XML which is used by the project which has meant rebuilding the tables and the Xapian indexes.

Following a request on the Open Literature mailing list, I’m looking at the idea of using Python’s NLTK to create some linguistic API wrappers around the Xapian search. It strikes me that these letters can be used to create a corpus of Dickens’s language where you can explore the language used in family correspondence (to his daughters and wife), to other authors (Wilkie Collins) and to readers. That is a longer project though in terms of building the relevant indexes.

I’m also looking at the idea of clustering a collection of letters to a correspondent and seeing what happens (for some reason, the current script is looking at Wilkie Collins). There is also a set of queries that one might run against letters discusing books and the publication dates to view the distribution. I’m working on these latter questions at the moment for intended release later this week but I do foresee it being delayed a while.