I’m trying to move them onto the web and make the useful, not only in terms of reading about them but also to make them linkable. I’m trying to get them out of being placed on a web site as Word or PDF downloads and move them into being web pages with comments. Drupal 7′s inbuilt book module is probably the way to go and is producing some really nice results in the hacking I managed on Friday. There is a certain pleasure now in that I began the hack at 8:30 and within an hour, I had a working document (albeit I wanted to mess around with the URLs to make the nicer and far more meaningful). It had comments and was generally felt to be good.

The next task was to work on a way of doing Frequently Asked Questions (FAQs). Having begun some of the work using the Frequently Asked Questions module, I decided it had to many issues for us (including not being able to control where the page was and it did not appear to play nicely wiht the Pathauto rewriting module), I write my own content type which we can manipulate via the Views module to create sets of FAQs. When I’ve got more time, I may come back to the module and try to help fix some bugs.

Whilst neither of these are finished items, it was a pleasant day hacking and creating, getting prototypes ready in a day. I’m taking this as a sign of increasing familiarity with Drupal. I do, however, need to find a morning to finish the Sugar SOAP integration module and tidy that up. Ideally I’d trying to find a way of integrating it with the current module to offer swapable backends.

I’ve also started looking at using Redis for caching again in a major way to ensure that various static fields of data, such as UK counties, can have a common reference to reduce data cleaning issues such as county begin written as co., co and county. I’m also looking at the issue of Linked Data and how to integrate the ideas into our current projects. For now I’m rereading Tim Berners-Lee’s guide, linked from the linkeddata.org website and formulating ideas and refining the ones I currently have.

Ambition might bet the better of me but at least I feel like I want to take all of this on and to try to improve skills and learn more. In the meanwhile, I have some serious hills to climb.

Update:  This post has got me rethinking the Open Correspondence RDF and Linked Data. The more I delve, the greater my sense of needing to rethink that part of the project and to complete the correspondence links. Most of them are there but need complete linking. I also need to look at the Python’s RDFlib and perhaps make better use of the Sparql qeuries and stores. I sense an evening or several of experimentation before a hacking weekend to resolve these issues.

The major thing that I have completed is the basic integration of Drupal 7 with SugarCRM Community edition. At the moment it definitely works with version 6.12 as this is what I have at the moment but I’m going to upgrade to 6.2. I do not see any issues regarding this as apart from having to remap one or two fields. I’m hoping, next week now, to split off some of the changes and to offer them to original project as patches so that the main webform2sugar project can bring them on board or not as they will.

In tandem with the data cleaning project mentioned last week, I am looking at caching data using Redis behind forms to offer fixed lists of data. Although we commonly use PHP, I am strongly thinking of writing the readers and document parsers in either Perl or Python. What I might do is to write some test scripts in both and benchmark them but also have to balance their handling on MS Word (probably largely the 2003 version rather than the 2007 one) and PDF documents across platforms as I will be moving them across platforms.

I’ve also been thinking about NoSQL stores for other pieces of data and projects which are being worked on. The HighScalability blog has a great piece on what to look under which circumstances for SQL and NoSQL databases.

The statement linked to is more than terse but I wait with interest to see what this new platform is and what it will do to “provide new benefits beyond JISCMail’s traditional boundaries”. I did hear mention of social networking in passing and integration with social networks was being discussed whilst I was there.

Further announcements to come…

This has long-term implications in terms of continuing to ensure that the data is clean and standardised. Given that one of the forthcoming projects is based on our technical documents and converting them from existing formats (when these are fully confirmed) into the , as yet unbuilt or designed, system. As part of this I’ve been looking at the Chris Gutteridge’s Grinder, a parser for getting RDF data out of Excel and CSV files. I was reminded of Grinder whilst reading his article about Linked Data at the University of Southampton in the final ever Nodalities. Whilst Grinder itself may not be of initial use, it does give me some clues about the possibilities of transforming the data.

The project also forces me to think about how the programme would run and I suspect off the command line. If this is a safe assumption, then it means that I need to get back to Perl or use Python. Much as I like PHP, I’m not sure it is a command line language. I know it can be run as one but it always make me nervous as I don’t really consider it a system administration or data munging language. In either case, Perl and Python mean another re-learning curve, especially Perl which I last use at JISCMail a couple of years ago.

A side project that I’ve been  looking at is the real-time data storage of feeds for later mining and use. I’ve been thinking of using Node.js (and actually starting something!) and Redis to run in the background. A little side something, methinks. It does mean me learning more about Node though and gives me something tangible to build. I’ve been having a little search around the Net and came across an older post by Marshall Kirkpatrick on the NTEN blog about realtime data whilst reading about event loops in Node on the Elegant Code blog. Of course, once it is stored, it must be processed to be useful but that is the next step.

After Book Hackday, I’ve written a post about beginning to develop the Open Correspondence toolkit for the Open Knowledge Foundation’s Notebook blog. I was also contacted regarding converting the TEI XML pages into JSON, which I am currently working on.

Once I’ve done some more work on it, I’ll post the code and more about it.

I’ve been working on another project which may or may not be open. It is certainly interesting but I am not sure I can say much more than that. I hope to have a blog post up soon about it but I am rather excited by it and its possibilities.

Meanwhile, the work project continues apace with some surprising outcomes for me. Following watching a video on Facebook’s architecture, I’m beginning to see certain parts very differently. I really do hope more on this but I’ve got some building to do and a bit more delving and reading that needs completion.

The current changes are:

• additional fields in the RDF endpoint.  I still need to do some major work to JSON and XML which I hope to do for the next update.

• a basic text search

• a basic set of geographic data in the collection

• better linking from the letters to the correspondent and geographical data (NB it is still incomplete)

• some mapping with Open Layers javascript.

• a Simile timeline (which is a bit slow at the moment).

Admittedly some of this is exposing work already there but hidden. However I’ve also been working on some unicode fixes to the underlying XML which is used by the project which has meant rebuilding the tables and the Xapian indexes.

Following a request on the Open Literature mailing list, I’m looking at the idea of using Python’s NLTK to create some linguistic API wrappers around the Xapian search. It strikes me that these letters can be used to create a corpus of Dickens’s language where you can explore the language used in family correspondence (to his daughters and wife), to other authors (Wilkie Collins) and to readers. That is a longer project though in terms of building the relevant indexes.

I’m also looking at the idea of clustering a collection of letters to a correspondent and seeing what happens (for some reason, the current script is looking at Wilkie Collins). There is also a set of queries that one might run against letters discusing books and the publication dates to view the distribution. I’m working on these latter questions at the moment for intended release later this week but I do foresee it being delayed a while.

I bumped into Eamonn Neylon and we went along to the Mobile Web session with Mike Jones from Bristol and the Molly project. They outlined the two main approaches (either via the various app markets or having a front end which caters for the different phones) and issues such as being sandboxed from the hardware layer at the moment. It would seems from them that you need to do both ideally, though development time doesn’t always allow. The session was slightly hijacked by the Python 2 versus 3 question and if Molly would ever use Python 3 but we gradually got back on track. The main barrier to entry would be the lack of standardisation so you need to target the platforms as well as the hardware issues.

I stayed for the Linked Data session which Chris Gutteridge took sort of control from the array of speakers. The main focus became notions of openness (as defined in the Open Knowledge Definition) and how it is perceived by the academic community bringing back up the ideas of attribution on the web. (An issue which is partly cultural.) The issue of clear licencing came up again as well but there does seems to be some clarification needed on the different models.

I did go to some of the lightning talks before lunch but they didn’t leave much of an impression this time (though Chris Gutteridge did plug his Q &D RDF Browser which I’m thinking of using). After an excellent lunch, I wandered into basecamp where I plugged in my laptop and worked on Open Correspondence whilst waiting for the session on the Linked Data API. I did spend a couple of hours work on it to fix some bugs and little things for the next version to go live (though discovered another one in places with some missing but it is not huge, just needs a couple of hours). Rufus Pollock and Jo Walsh popped by so we managed to catch up and do some hacking. Rufus suggested using Flask which I think I’ll use for some smaller projects in the future (and for some reason Backbone was mentioned but not sure how).

I went along to Chris Dollin‘s talk on his eLDA library and how the Linked Data API works. It seems like an eminently sensible solution to removing the complexity of semantic technologies from the user and to make it easier to use. It is certainly something which will be useful to get my head around completely.

The day, as suggested by Mahendra Mahey, was definitely more useful when doing something and just cracking on with it. We need more days like this as it provides a collegiate atmosphere to try new things and take a look at different technologies which might not appear on the normal radar. The friendly atmosphere was great as well. I’ll book both days off if it comes around next year.

I’ve finally done the first pass of this update which I’m hoping to make live soon once I’ve updated the controllers and re-checked the other data improvements. Whilst it is not perfect, it is a lot better than it was. I think that the next week will be spent going over the endpoints and the Pylons controllers so that the data is cleaner than at present and correctly linked.

It has been a useful exercise in that I’ve started rewriting the parser for the letters (an ongoing large job I was thinking of doing when I come to the next set of letters) and putting some of the earlier thoughts into place.

Once I’m happy with these updates, I’ll update the site which does mean rebuilding the databases and endpoints. However once it is done, it should be a lot cleaner  and I can then start looking at the correspondents and linking into other data sources like dbpedia.org. I think that the first task though might be to restart work on the clients that I had been putting together  as a basic development kit.

Firstly, I need to work out a more exact way of mapping the data in the database or flat file. I think what I really need is to use something like:

• place
• address
• city
• latitude
• longitude
• description
• url

The data that I have is not quite as granular as this. Yet. When I’ve done this, I need to build the mapping so that if a place is entered, say Hotel Meurice, Paris, then I can return the details and latitude / longitude to render an Open layers map. That’s almost the easiest bit really.

The second issue is the difference in names. Over time and in the heat of writing, names can change subtly. For instance Gads Hill Place, one of Dickens’s homes which is now a school. In the letters it is referred to as

1. Gad’s Hill Place,
2. Gad’s Hill Place, Higham
3. Gad’s Hill

It can also be known as Gadshill Place or Gads Hill Place. I need to find a way of differencing the terms. Firstly I need to develop a way of checking inside a term and then returning it if it is a new terms or returning the mapped version if it matches a term. Secondly I need to fuzzy match the strings so that any near differences (using the Levenshtein edit distance) can be checked and either ignored or exclude the term.

These issues will also affect the correspondent code which is being created. I suspect that anything with names will have the same issues. For  instance, Wilkie Collins is known in the letters as Mr W Wilkie Collins and Mr Wilkie Collins. In the current implementation of the site, these are two different entities which is clearly wrong. They are the same entity but there is a subtle difference which is not accounted.

So to deal with this, I am going back to the parsing library and building these in instead. Whilst it is a slower way of dealing with these issues, it provides a chance of doing any necessary information and site re-thinking.

As part of this, I downloaded some TEI guidelines from the California Digital Library to use to build the base metadata export. Ideally what I’m hoping to do is to create the data as a Python dictionary and then reformat into HTML, HTML & RDFa, RDF, JSON or XML. It should allow me to export the same data for each type.

I’m sure at times I’ll wonder what I started but it needs doing if the site is to accept more authors. After that, back to search.

On a separate note, I have also done some work on the Arts Funding search. I’ve given it a re-skin and used the Accordion widget from the JQuery UI. It also has some more search options built in so that the data can be searched by date and amount as well as political constituency and art form. The search needs to take in some arguments such as < or > or equals in the amount but that can come. I’ve been reading Jenni Tennison’s post on the data.gov.uk blog to best expose the data using Linked Data.

Whilst writing this post, it occurs to me that whilst Linked Data is an awesome way of exposing data, useful search is still an important part of any content driven website. As blogged before, I have implemented an early version of a Xapian search. As Tim Bray has noted, advanced search might have a smaller use but it is more likely to be used by the heavier users so deserves to have time taken on it.

I’ve gone back to the some of the work that I was doing for the first version of the site way back last year. As part of the move to Linked Data, I’ve been working on a URI for places and books. Places, asn oted in previous posts, has come together and is just in need of some tidying up. I’ve managed to create an index page from the RDF endpoint using rdflib to parse the triples looking for the geo: namespace and then putting the items into a set to remove the duplicates. This needs changing as sets are unordered and I’d like the page to be ordered so that a pace can be found quickly. Perhaps a better option would be to place the raw data into a dictionary and cast to a list to sort at the last moment (or more simply sort the keys in the dictionary…) and then to remove the duplicates such as Gad’s Hill which is analogous to Gadshill. Both are used but refer to the same entity, so I need to do a difference pn the string (probably using difflib or a variant)  to identify the changes and clean up the URIs.

With the books, I had created a table of the publication dates and the titles, so all I need to do is to map the book’s variant titles, such as the “The Adventures of Nicholas Nickleby” is better known as “Nicholas Nickelby” or plain “Nickleby” in the letters. It might be easiest to put this into a dictionary at the moment rather than another table and to call that. I would also need to get some sort of introduction (and perhaps in the future create an Open Dickens site for the novels).

I’m sure I can do this in a few hours and to get it working. Must make the time now I’ve had a small break.