A quick Google search brought me to this post by Roy on building an HTTP service using Tornado. I am fairly new to Tornado but have been looking at it for an experiment at work. However I have been using Flask for other projects such as a quick and dirty RDF browser largely based on Chris Gutteridge’s PHP browser. (At some point very soon, I need to get back to this but other projects are slightly more pressing at the moment.)

After a quick upgrade to version 0.8, I have managed to put something together a little like Roy’s script but I’m hoping to go further and add a storage layer before tidying it all up.

Unlike Roy’s script, I’ve pushed the Tesseract and file handling code outside of the server. In the long term, I’d like to split out the file handling and storage facilities from the web server which means looking at the storage. As a quick step, I’ve popped in a link to a MySQL database but a far better option would probably be a NoSQL database like CouchDB or similar. I suppose a Key / Value store like Redis could be used as well (Redisfs apparently does something like this) as a back end. I’m keeping options open.

I do have a temptation to use RabbitMQ to notify various workers that a file exists which suggests that if I’m hoping to use this as a book scanner back end (discussed at the Textcamp event in August), then I need to add in an automated set of scripts which reads a directory and deals with the file and moving, storing and scanning them. Perhaps Tornado might be a long term answer but realistically it is not needed for a test project.

Also, Tesseract will need some training as I’ve discovered this evening playing with some newspaper text and seeing some of the results. As one of the reason I began this was to store old fanzines and newspaper articles which I’ve stored for research but are now degrading, that might be a problem.

Either way, this is a way of moving ahead with the book scanner conversation and building something small to scratch some itches.

One of the matters that did concern us was the word “data” and what this might mean to literature and humanities. One assumption that we had was that it perhaps had a more defined meaning to scientists. But what is data to humanities? Is it the manuscript, the notes, or the published work? We decided that ‘Work’ might be a better word for the overarching principle.

One of the issues that is important is re-use and subsequently closing the re-used work down and making it non-open. The major party that we had in mind was Google Books. Whilst they are making good and admirable strides in the digitising of out of print works but there is no API or metadata store that can be used to mix up the data or to mine it in any other way. Effectively we end up where we started: with a technically open text tied up in ways that cannot be re-used.

Re-use and re-mix are extremely important within digital humanities. Influence and building on works are central to movements like Modernism and also ensuring that works and authors are accessible. Works are adapted and take on their own lives or segue from such moments.

The final major point was that citations and the underlying cited text should be open. Whilst the core of the principles are about the work and ensuring that it can be worked on, a fair amount of work goes into notes and annotations to the text (such as the great Annotate It tool) and these provide a meta work for people to build on. It is vital for debate that these are not put into a closed arena, not just for the sharing of notes but also building on the notes. They might also be put together into a new work or an annotated version of a work put together to build upon the work with communal notes.

This does represent a step forward in open literature and digital humanities. I really hope that debate does start and that these can be developed and make concrete.

Wordcloud of Defoe's journey taken at Textcamp

Using the Many Eyes Word Cloud, we can see that Scotland is unsurprisingly the largest item but also Lord and Earl are also popular, suggesting that he stopped with or met the aristocracy rather than just travelling randomly. Dave Flanders and John created some cool visualisations using the tool which allow you to follow words in the text and to see which are the most linked to words (using bigrams I would suppose) in a tree fashion. It is certainly something at I will be looking up later for “quick win” visualisations.

One of the intriguing projects that was suggested was building our own DIY bookscanner using links currently stored on the Textcamp 2011 wiki pages. I think that Dave Flanders might be organising a hack weekend to actually build the machine for real use. I find it interesting but thinking that it would be cool to also see if can be built at home or using iPhone / Android OSes which also entails a software hack, unless an app already exists. That is something to explore later.

Mark MacGillivray, of OKFN and Cottage Labs,  and Brian Hole of Ubiquity Press, spoke about Open Access and making scholarship open but also retaining its rigour. Using Open Access, we should be able to share the data, the ways of interpreting it and and the final interpretation which is published.

The science community has been doing this for some while and things like the Panton Principles and Science Commons are showing the way. One of the ideas was to write a handbook for how to use openness in literature and that it is something that we need address and build on. We ought to write an open guide / manual and build on / develop the Panton Principles where necessary as a core set of principles to work with.

Having days like Textcamp and Book Hackday are extremely useful to think about this and to work on the ideas. It is easy to get into echo chambers of mailing lists and blogs, we need these events to meet new people, be challenged to explain ourselves and to either build on the day or go away with ideas to test and try out. The day has excited me out using word clouds again and doing a bit more work on them as a tool to make them useful. It has also got me excited about book scanning and doing some hardware hacking (which I’ve not really done) before.

Running the Pope essay through Wordle makes me excited about testing what we can do with the ECCO TEI documents that John Levine  links to. Can we hyperlnk to other texts, author and events that are mentioned in it (not just with the annotator tool but in generated HTML) or use HTML 5 to embed audio links to further discussions or pronunciation (for example Byron’s Don Juan which has been argued as pronounced “Jew-an” rather that “Hwan” and the arguments for and against).

Perhaps that gets to one of the issues that arose in the break-out discussions in the kitchen. After the lightning talk about digital publishing, there seemed to be an argument about whether current digital publishing was really pushing the boundaries or flailing around. I do think that it has some real benefits for niche publishing but these have not been fully explored. The model will need to change and perhaps become more open in those senses, perhaps linking the raw data to the interpretation earlier to allow the relevant community to peer review the data earlier. Just a suggestion. There are two distinct communities, the top-down business layer and the grass roots layer, activists, data developers and so on. Both would appear to have broadly similar aims but how to put them together  in a useful way for both to learn. Don’t get me wrong here as I believe I’m at the grass roots layer, but I think that both sides do have a dialogue which could get around the issues that the music and film industries have found themselves in, i.e. confrontation. We are here to disrupt and make.because we are passionate.  We care about the industry. Publishing is an industry which needs to change and transform itself. Put the two together and there are ways of moving forward. My hope is that in future events, we could get some more publishers along to the event.

The other important thing is that these conversations carry on afterwards. The round table discussions where great as were the break-out in the kitchen ones but they need to carry on or we create our own echo chamber which reduces the value of what happened yesterday.

Whilst I did not do as much coding as I wanted to yesterday, I met some new people and caught up with colleagues. The fact that organisations such as JISC are supporting events like this shows their underlying importance and use to the community. We’ve started, now we need to carry on by chatting, blogging, sharing and doing more of these events.

Whilst  listening (and remembering how much I had enjoyed it when I read it years ago), I began thinking about trying to expose it as Linked Data so that the book’s publication detail could be linked into bibliographic data, the actor’s details could be linked as well so that you could see who appeared regularly, or which companies were regularly used by the BBC to create these dramatisations. If the bibliographic detail could be suitably developed, then you could also query the publication dates and authors.

I think that might well be a future small project to start doing (perhaps one evening). I guess it could be a script to scrape the relevant pages, query dbpedia (since I’m using PHP on this server, using ARC) and return a relevant form.

I’ve been working on two things for the new version: timeline and full text search. Xapian has now been installed as the search engine but the move has meant that Ineed to make one or two changes which means that it is not live yet. I hope to have the issue resolved shortly as it is not huge really.

There is also a nascent timeline of all of Dickens’s letters which still has some issues, like taking its time to load loading as there are around 1000 items. I’ve a feeling that Rufus might be looking at this when he has a moment.

Last week, I posted the next steps that I’d like to take with the site to the Open Knowledge Foundation help and open literature lists. The next thing is to start exploring geographical information and to expose that data.

The upshot is that it is time to re-look at the parsing methods and to really beef them up. They sufficed for us to get the project ported from the original PHP and to get the site up  but now I think I have to relook at each method, write more unit tests (and combine them into the openletters tests as they are separate). As the project gets bigger, the value of unit tests becomes much more apparent in terms of ensuring that we have not broken anything which far outweighs the time taken to set them up. It is habit that I need to force myself to continually do when developing. I’ve do this for some of the systems that I’m building at work using PHPUnit).

The second point that comes from this is storing the data. Currently the site reparses the data for the end points but we’ve been talking about using CouchDB and possibly using GeoCouch for the next version.The idea is that we can then store the data and then transform it to the correct format when requested.

In part I’ve decided the only way of finding out how the site might be used is to, well, dog food it. So I’ve also started writing a Java client using Jena for Sparql to retrieve lists from the rdf endpoint and to represent them in XML, JSON or HTML. Currently the SPARQL query is built (though need schanges due to last night’s move as the RDF endpoint moved) and I need to do to complete the change from a List<QuerySolution> into a readable form like XML and  so on. The idea is that it will come as a JAR which can be packaged onto the class path of a WAR or another system.

I’ve also got a Python script to cluster the letters on the go as well which I will commit to the Python bitbucket repo once I’ve got a bit more done on it as it currently only builds the intial matrix so I need to visualise the data next.

What I’ve been thinking about is software re-usability and I realised that quite a few of my queries are very similar in terms of looking for abstracts and influences on a text. I’ve started collecting various scripts together and putting them together to form a small library to re-use across texts to develop over time.

I’ve been writing some scripts which wrap around the ARC toolkit and query dbpedia. It still has a fair few bugs but I need to spend some time getting rid of them. At the moment, the abstract overwrites all the text if you scroll over the title or Milton’s name.

Whilst PHP might not always be my language of choice but one of work, I’ve started porting some of these ideas across to Java and Python. As part of this, I’m hoping to finish some stuff I’ve been working on for some command line scripts and tools for the Open Correspondence project which will hopefully feed into the project.

I still have not read the Reflections yet but I thought it might be fun to get the Gutenberg text, strip it down to the text itself and begin building a tag cloud as an exercise in text mining. What I’m trying to achieve is a visualisation of the text and the keywords. That will probably take me a couple of attempts in cleaning the information up and making it relevant.

Having done that, I’d like to make the cloud relevant by clustering words together and linking them into the text or search. That is one of the things that really annoys me with some tag clouds. Pretty pictures but no linking. I always think of it as why bother? Seriously, time is taken to mine the data, then visualise it but not to make it relevant or link it into the source material somehow.

For now, I’m going to leave it as a pretty picture version (sorry…) as I try to make it more relevant and release bits of it as I get it done.

Tag cloud of Burke’s Reflections on the Revolution in France.

On the todo list is linking the words to either a search and text and to contextualise by creating a corpus of words.

This follows on some stuff that I was doing a while ago to explore how make tag clouds and then use them. I was reading Jim Bumgardner’s Building Tag Clouds in Perl and PHP (OReilly, 2006). Lots to do but something to chip away at.

Update: Updated the Prospect link which is now available and corrected a spelling error

The soap server is largely debugged and ready for more extensive testing on the server and the back end has now been rewritten to capture more data. I cannot help feeling that it will change once more services go online to scale more efficiently but right now I don’t have the expertise to do it. I’ll get there.

On a different tack, I’m back on the accounting project that I was on several months ago and making some headway in that. Its grown since I was last involved in it but nothing that a decent set of specs and roadmaps cannot solve in terms of making it manageable.

I’ve been thinking about my next book project which is on the New Weird and genre over the last 15 years and wondering how to use dbpedia’s influencedBy and influence terms in terms of showing how writers influence each other over a century. I’m tempted to put the data into a large rdf sheet and then use javascript or PHP to transform it into JSON to see if you can use the Simile timeline software usefully or if I need to find / write something more appropriate. It does have to wait for me to finish the current book.

I forgot to link to the Open Correspondence blog post on the Open Knowledge Foundation’s blog which was posted a few days ago.

One of the next things to do is to reformat the dates into human readable format. I had thought of using Python’s datetime strftime to reformat the date from its current ISO format (YYYY-MM-DD) into day, month year. Unfortunately, the method states ” years before 1900 cannot be used.” A slight cramp in the plan. However there is an Activestate recipe by Andrew Dalke which might do the trick or at least point me in the right direction. It is one of the things to be tidied up at some point.

It is a good feeling to have the site running now. The next task is to write the tests and then  to refactor the code. It is very PHPish and needs to be made more Pythonic. I’ve got an idea for trying to create a dendrogram around the textReferred element and to discover the letters and correspondents around the books that Dickens was writing. One of the tings is to continue loading the other volumes of Dickens’s letters into the site. So version 0.2 is a little way off but the light at the end of the tunnel is not a train this time.

Workwise has been a little hectic. I must make some time to write a method to allow our admin team to resubmit applications. Like so many things it is a balance between a five minute job and the two hour ones that need to be done. The major job for the week though was getting the automated printing working.

One of the jobs that admin do is to go through each client and create the packs for them. Using HTMLtools, I’ve managed to compile the html into PDF and then convert the PDF into a PostScript file for a printer. I’ve managed to use the Line Printer Remote protocol to send the job to the printer. It is a simple enough command:

lpr -S <ip address/name of printer>  -P <name of print job> (-o <optional -o 1 sets file to binary>) <name of file>

Windows doesn’t appear to support the full protocol but enough to be useful. The -o switch appears to only define whether the file is binary or not rather than specifying the paper type and so on. Annoying but it can be got around.

Anyhow it got me thinking about other ways of using commands to explore how texts can be converted and changed into useful objects. It brings me back to the use of psbook for printing but how to make it useful for an average user who does not necessarily want to run various commands. Having had a conversation with my friend Darren Nash ,editorial director of Orbit books,  about the future of publishing; he opined that small presses would come to the fore. I think, certainly in genre that this is correct. It would be interesting to see how existing tools could be used towards these ends rather than constantly re-invent the wheel.

Now that the first version of letters is out the way, time to go over other projects. I’ve got a yen to try and create something from Milton’s Areopagitica, appropriate I think as it is a cry for free presses.