I’ve still got a fair amount of reading to do but this is the sort of project that makes me rethink the Open Letters project and how it could be used by other sites. It has also given me food for thought for work as well.

Still working on the accounts project which keeps unravelling a series of underlying problems. Most of them we know about but they appear in all sorts of odd places.

Assuming the world doesn’t fall on my head next time I’m in the office, I’m going to try and spend the day at home on a “Fedex” day. I’m taking the notion from an issue of Wired where they were talking about different ways of working and Atlassian mentioned “Fedex” days where you spend a day building a prototype. What I’d really like to get prototyped is the service bus / queuing system. So fingers crossed.

The impetus came from updating the disaster recovery documentation and writing the first department of the service status documentation (which I wrote after getting the last bit of debugging finished). I know that documentation is not everybody’s favourite thing but I find it useful in rethinking the system and making sure it fits together.

I’ve made time to rewrite the load function for Open Letters. I’ve got the document building the letters in XML and written a rough upload script. Next task is to rewrite the main.py script, test the XML loading and then finished tidying up the initial document.

I’m also looking forward to Textcamp so it’ll be great to get the load finished (as it normalises the function) and get on with doing a presentation for the camp.

I’m also coming to end of writing my book on children’s fantasy. Whilst not technical in an IT sense, I’m thinking of the next project on the New Weird and how to use IT to visualise influences and timelines. The one that worries me is archiving necessary web pages for the research which I need to look towards as I’m not sure whether it is technically illegal.

One of the next things to do is to reformat the dates into human readable format. I had thought of using Python’s datetime strftime to reformat the date from its current ISO format (YYYY-MM-DD) into day, month year. Unfortunately, the method states ” years before 1900 cannot be used.” A slight cramp in the plan. However there is an Activestate recipe by Andrew Dalke which might do the trick or at least point me in the right direction. It is one of the things to be tidied up at some point.

It is a good feeling to have the site running now. The next task is to write the tests and then  to refactor the code. It is very PHPish and needs to be made more Pythonic. I’ve got an idea for trying to create a dendrogram around the textReferred element and to discover the letters and correspondents around the books that Dickens was writing. One of the tings is to continue loading the other volumes of Dickens’s letters into the site. So version 0.2 is a little way off but the light at the end of the tunnel is not a train this time.

Workwise has been a little hectic. I must make some time to write a method to allow our admin team to resubmit applications. Like so many things it is a balance between a five minute job and the two hour ones that need to be done. The major job for the week though was getting the automated printing working.

One of the jobs that admin do is to go through each client and create the packs for them. Using HTMLtools, I’ve managed to compile the html into PDF and then convert the PDF into a PostScript file for a printer. I’ve managed to use the Line Printer Remote protocol to send the job to the printer. It is a simple enough command:

lpr -S <ip address/name of printer>  -P <name of print job> (-o <optional -o 1 sets file to binary>) <name of file>

Windows doesn’t appear to support the full protocol but enough to be useful. The -o switch appears to only define whether the file is binary or not rather than specifying the paper type and so on. Annoying but it can be got around.

Anyhow it got me thinking about other ways of using commands to explore how texts can be converted and changed into useful objects. It brings me back to the use of psbook for printing but how to make it useful for an average user who does not necessarily want to run various commands. Having had a conversation with my friend Darren Nash ,editorial director of Orbit books,  about the future of publishing; he opined that small presses would come to the fore. I think, certainly in genre that this is correct. It would be interesting to see how existing tools could be used towards these ends rather than constantly re-invent the wheel.

Now that the first version of letters is out the way, time to go over other projects. I’ve got a yen to try and create something from Milton’s Areopagitica, appropriate I think as it is a cry for free presses.

The first half was spent frantically converting html pages into PDFs using Verypdf’s HTMLtools server product. All in all the manual is very helpful and the test server could be set up quickly. It might have helped the other end if I’d remembered to break the file up for printing but that turned out to be a 10 minute jub to put back into production. The next task is to transfer it from the test server and onto the production one but that’ll need to wait for networking to tweak it a little.

I spent some time refactoring the call recordings archive. For some reason the archiving solution that I hacked up in November decided to start failing in March after it was changed. Despite being put back to its original state it never quite got back to working as it did. I’ve been trying to tweak it ridon and off but never found the time to complete it. I finally just made the time on friday afternoon to look at it properly. I’d been thinking about item based filtering after reading the first chapter of Toby Segaran’s Programming Collective Intelligence. (On the back of this, I think I’ll be buying his Beautiful Data at some point.)  Although this is not really an intelligent programme as such, the techniques have shown some real promise in the hurried tests. Using a Redis datastore, the percentage of found recordings is way up. Fingers crossed for Monday morning when I can see what the scripts run over the weekend. I also spent some time simplifying the matching algorithm so that I didn’t have to account for so many edge cases when dealing with time.

It seems that we are approaching some sort of real-time status update systems at work. I’ve sort of been arguing for this for a while to remove the bottlenecks of having each system dependant on another one. One of our suppliers is sending us XML data so I’ve been playing with Xpath 1.0 (since Xpath 2.0 apparently isn’t directly supported by PHP but there might be a way of passing the data to Java which adds unnecessary overhead) to extract the relevant values. Anyhow the core is running but I still need to fully test it and add in security.

I’ve also been asked to design and implement a queueing system for the main internal server. I’ve run up a quick high level overview but the detail still needs to be worked on. I’m pushing it back to June so that I can slear the decks of the older projects that are still on the board.

I had a chat with Jonathan Gray, a sound guy who does far too much, about digital humanities ideas. We’ve agreed to keep closer contact with each other about the area and to encourage each other into actually doing stuff (I have half a moleskin of ideas – time for more code, less talk then).  He proposed the Bibliographica idea in January and the team wrote a blog entry for the Open Knowledge Foundation blog. It is an idea that I’m looking forward to playing with and trying to embed data from. (http://bibliographica.org/)

One of the things that I’ve been thinking about though is increasingly when we do research, we store  web pages, blog entries and so on. Whilst there is way of recording these in a footnote (http:example.org accessed on <insert data> type thing), there does not appear to be a way of building a local archive of these with the relevant metadata for later retrieval, Don’t know about anybody else but I’ve got a fair few pages dotted around my hard drive for projects and I’d like a way of storing these properly and to be able to integrate them into bibliographies or research notes. I know that there is WARC format (Library of Congress link and the WARC tools Google code project) to play with so need to make time to do that.

I had a mini-hack on the Open Correspondence project last Sunday intending to update a couple of pages and got a little more done than that. The database needs rebuilding but the purl reference (http://purl.org/letter) now points to the schema. It is so close that I can’t wait to actually start hacking the data. Time to do the last little bits like tidy up the parser, use the weaving history API to embed a timeline and start using JENA, ARC and Chris Gutteridge’s Graphite library which worked out of the box (but as yet I haven’t entirely used it for much yet).

Goals for this week are to finish the Open Correspondence bits, update the trac instance with the various ‘todo’s, write a blog post for the Open Knowledge Foundation for Open Correspondence, do some major testing this week at work on various XML exports and imports. I should just be about caught up then. With any luck…

One of the projects that I’ve been working on is the temporary storage of information. For one reason or another, one of the workers has decided to occasionally throw a fit and not do its job properly (on top of a connection that appears to fail at odd times). What I really needed was a temporary store to save the parsed information so that if something failed, we didn’t loose everything. To that end, I’ve started looking at Redis in more detail and started using the Windows build of version 1.2.1 (available on aspninja.com) with the Rediska library. At some point I’ll sit down and compile it on my laptop under Cygwin to get the latest version.

I ended up using the PEAR version of Rediska and managed to get it up and running fairly quickly. One of the things that I needed to do was to call a new instance of the list that I was creating in each method, having split the set and get methods into two workers. The speed of Redis is fantastic and the server happily runs on the test server caching the data and allowing another worker to load into a copy of the MySQL tables that it will eventually update. I found the Rediska library really easy to use and I’ll be using it for various projects at home to do some processing rather than using MySQL all the time. Simon Willison has a post which links to a tutorial on Redis that I found extremely useful and encouraging in finding more about the server in future.

I’ve been working on the RDF exports for the open letters project which are yet to go live. The main job has been making sure that the exports validate using the RDF validator and pulling in the data. A future task is to finish tidying up the data but I’m trying to get the letter html template figured out. Since Python isn’t the main language that I know use (work is entirely based on PHP), I’ve been taking a look at the Open Shakespeare code and found that RDFa work that I worked on a year ago and completely forgotten about. It would be good to get RDFa into open correspondence but I think that is a later task. Main thing is to complete the initial port. I managed to get the www.purl.org/letter forwarding to the site but need to get a schema page up and the purl correctly referring to the right page.

One of things that I’ve been trying to play with RDFlib on Windows. I built it successfully on my last laptop (Windows XP, Cygwin) but for some reason version 2.4.2 would not build on Vista, even under easy install. I’ve been trying with the version 3 (which has just been released on may 13th according to the news group) and apparently the rdfextras project has a pure Python version of the Sparql parser which was failing to build. I’ll be trying that once the current work on open correspondent as been completed to explore what we can do with the data.

Ben O’Steen talked at the Open Knowledge conference after me and one of the things he talked about was the psutils package. I’ve found it on Cygwin and downloaded it so it would be good to have fun with that one or to find accessible Windows ports for people who don’t necessarily want to download Cygwin.

Simple Ontology for Relationships in Texts

Text namespace

austgate.co.uk/ontology/text

Definition: An ontology which allows for the linking text items, such as letters, together. It extends and complements Dublin Core (DC) and Friend of a Friend (FOAF).

Terms

Appearsin

The term is used to denote a work in which a character appears. For example:
Dear Alice,

As you may know I am coming to the end of the latest draft of the Ponsonby diaries. Bob Ponsonby is making his way across the marshes…

The character Bob Ponsonby could be referenced as text:Appearsin to denote his appearance in the work. This allows queries to find documents where the characters from a work appear, rather than just individual characters. It would usually be considered as a collection of text:Character references.

Character

A fictional person who is referenced in the text. This element is used to disambiguated between fictional and non-fictional characters. Non-fictional, i.e. real people, are denoted by foaf:Person. Character is a subset of foaf:Person and is intended for fictional people. For example, in a letter from an author to an agent, the author may describing their latest project.

Dear Alice,

As you may know I am coming to the end of the latest draft of the Ponsonby diaries. Bob Ponsonby is making his way across the marshes…

In the example, Alice is a real person and could be denoted as such by using foaf:Person but Bob Ponsonby is equally a name and a person. Since he is fictional in this letter, he could be denoted as  text:Character in any RDF representation to allow users to link documents where the character is mentioned.

<text:character
rdf:ID=”http://austgate.co.uk/Dickens/characters/pickwick”>
<foaf:name>Mr. Pickwick</foaf:name>
<text:appearsin
rdf:resource=”http://austgate.co.uk/Dickens/works/pickwickpapers” />
</text:character>

Correspondent
This field denotes the correspondent of the letter.  It is a subset of foaf:Person as it should denote a real person. (However it is perfectly possible for a fictional letter to be written and in this case it would perhaps be inappropriate to use foaf:Person).

textReferred
This refers to a text (book, verse or similar) which is referred to in the letter being serialised. It is intended to allow the building of graphs between the letters where a text is being referred to so that a graph can be built of what an author was doing or thinking about a text around the time or after writing the text. It is designed to allow for some contextualisation of the referred work. It could also be used to build a reading list, possible influences or forgotten works that the author was aware of at the time.
Work

The term denotes a type of text, in this case a book. It would be a collection of Dublin Core terms.
<text:work rdf:ID=”http://austgate.co.uk/dickens/work/pickwick”>
<dc:title>Pickwick Papers</dc:title>
<dc:author
rdf:resource=”http://austgate.co.uk/dickens/people/CharlesDickens”>
<dc:publisher>Chapman and Hall</dc:publisher>
</text:work>

I’m still working on applying some of this to my letters project (which sort of came about because and from the curiosity about the idea). Many thanks to Brian Matthews of the e-Science department of the STFC but any mistakes or oversights are entirely mine.

the data age is less about the raw size of your data, and more about the cool stuff you can do with it. Now that there is so much data, it is time to unlock its value.

It seems fairly straight forward given the lower barriers to growth and tools to create and access data.

There are issues with this such as learnng how to best leverage these for the user and to gain most benefit. It’ll certainly be an interesting time and Cross identifies a few technologies and ideas which may or may not gain currency but will spark debate nonetheless.