I was helping out some colleagues by testing some existing code this week. not because I do not have any thing to do but because they are woking on something large which required some help testing and bug hunting. I worked on the test plan with my manager and extended it when I was looking at other parts of the site. This was the moment when I lost my testing “cherry” as it were and fired up the selenium IDE in Firefox.

Using its simple recording and playback facilities, I was able to reconstruct our testing steps and even add to them as I saw that assumptions had been made in the original document.

The simplest way of creating tests is to fire up Selenium and to go to the beginning of the tests that you want to run. Then click the record button and it will watch you click around the site and record a series of steps. When you click record again to stop it running, you can get it to run through all the steps or just one step using the playback buttons.

If you need to add a step, you can use the really easy interface to create or delete the steps and the actions that must take before re-running the tests that must be taken. It does take some time to set up the tests but that just be weighed up against the costs of running tests manually and perhaps missing steps.

Once the tests are complete and running, then you can save the tests using the file option and then create  a folder of them to run as required. Alternatively they can be exported into various formats including jUnit and PHPUnit.

The next step will be to add unit tests into code and try to backport some tests in existing code. Set up time is outweighed by the saved time later. I suppose a future evening project will be to put this together with a Selenium server and pull code from a repository.

The other tasks has been writing documentation for recently completed projects and some ongoing and planned ones. Having written up some user stories for new ideas and also giong back over old code has made me re-evaluate some of the code and to consider it as part of something organic. It is less of a chore now and more a help to myself and colleagues so that, when the paperwork and wikis are up to date, we could in theory have a break and if something happened, one of us can dive in. Alternatively we can make out iterative process better and stronger since we are not trying to rediscover the process of something by following code until we stumble across the answer.

I did have an interesting time with HTML5 geolocation which I need to look into more deeply. Having got some code quickly working to show a position on a map, I found some dead area in Oxford whilst showing somebody else the app. I am assuming that the geolocation runs off masts and the GPS system but it does raise a caveat to the enterprise of using pure HTML5. Need to put the code  onto a page and wander around the town with my phone pne Saturday.

Open Correspondence

Development on the Open Correspondence project has been slow to stalled for a while. I have been doing bits and pieces but sitting down with Mark McGillivray of Cottage Labs and the Open Knowledge Foundation, brought some clarity. Recently the Textus project has been announced and I have been talking with the developers to put the data onto that platform. It seems to me that it is better to pool resources and to contribute where I can. There are parts of the existing project that I like and others that need more work to make me happy and it seems right now to move onto the developing platform.

Textcamp

At Textcamp last September, one of the sessions covered DIY Bookscanners (Austgate post on Textcamp). One of the actions on the Textus wiki was OCRing text. I have posted previously about playing with Tesseract and seeing this, I emailed the humanities-dev list to explore the possibilities. To this end, I have volunteered to work on the area and will write a blog post about it There is already a large amount of work that exists, so  I am perhaps not developing anything new. However it would, I think, be interesting to develop a stand-alone system that is flexible and downloadable. Like other OKF projects, it will be a Python project but also be a hardware project to try and extend some of the existing projects.

Other Bits

I’ve been working on an indexing project which appears to be coming together quite nicely. Hopefully I’ll be able to say some more shortly but it depends on a conversation that has yet to be had.

Next week, after a break, is a return to work and to data. The Dev8d conference provided me with some ideas and clarity on one or two things, so time to put them into practice.

I’m trying to move them onto the web and make the useful, not only in terms of reading about them but also to make them linkable. I’m trying to get them out of being placed on a web site as Word or PDF downloads and move them into being web pages with comments. Drupal 7′s inbuilt book module is probably the way to go and is producing some really nice results in the hacking I managed on Friday. There is a certain pleasure now in that I began the hack at 8:30 and within an hour, I had a working document (albeit I wanted to mess around with the URLs to make the nicer and far more meaningful). It had comments and was generally felt to be good.

The next task was to work on a way of doing Frequently Asked Questions (FAQs). Having begun some of the work using the Frequently Asked Questions module, I decided it had to many issues for us (including not being able to control where the page was and it did not appear to play nicely wiht the Pathauto rewriting module), I write my own content type which we can manipulate via the Views module to create sets of FAQs. When I’ve got more time, I may come back to the module and try to help fix some bugs.

Whilst neither of these are finished items, it was a pleasant day hacking and creating, getting prototypes ready in a day. I’m taking this as a sign of increasing familiarity with Drupal. I do, however, need to find a morning to finish the Sugar SOAP integration module and tidy that up. Ideally I’d trying to find a way of integrating it with the current module to offer swapable backends.

I’ve also started looking at using Redis for caching again in a major way to ensure that various static fields of data, such as UK counties, can have a common reference to reduce data cleaning issues such as county begin written as co., co and county. I’m also looking at the issue of Linked Data and how to integrate the ideas into our current projects. For now I’m rereading Tim Berners-Lee’s guide, linked from the linkeddata.org website and formulating ideas and refining the ones I currently have.

Ambition might bet the better of me but at least I feel like I want to take all of this on and to try to improve skills and learn more. In the meanwhile, I have some serious hills to climb.

Update:  This post has got me rethinking the Open Correspondence RDF and Linked Data. The more I delve, the greater my sense of needing to rethink that part of the project and to complete the correspondence links. Most of them are there but need complete linking. I also need to look at the Python’s RDFlib and perhaps make better use of the Sparql qeuries and stores. I sense an evening or several of experimentation before a hacking weekend to resolve these issues.

The major thing that I have completed is the basic integration of Drupal 7 with SugarCRM Community edition. At the moment it definitely works with version 6.12 as this is what I have at the moment but I’m going to upgrade to 6.2. I do not see any issues regarding this as apart from having to remap one or two fields. I’m hoping, next week now, to split off some of the changes and to offer them to original project as patches so that the main webform2sugar project can bring them on board or not as they will.

In tandem with the data cleaning project mentioned last week, I am looking at caching data using Redis behind forms to offer fixed lists of data. Although we commonly use PHP, I am strongly thinking of writing the readers and document parsers in either Perl or Python. What I might do is to write some test scripts in both and benchmark them but also have to balance their handling on MS Word (probably largely the 2003 version rather than the 2007 one) and PDF documents across platforms as I will be moving them across platforms.

I’ve also been thinking about NoSQL stores for other pieces of data and projects which are being worked on. The HighScalability blog has a great piece on what to look under which circumstances for SQL and NoSQL databases.

The statement linked to is more than terse but I wait with interest to see what this new platform is and what it will do to “provide new benefits beyond JISCMail’s traditional boundaries”. I did hear mention of social networking in passing and integration with social networks was being discussed whilst I was there.

Further announcements to come…

This has long-term implications in terms of continuing to ensure that the data is clean and standardised. Given that one of the forthcoming projects is based on our technical documents and converting them from existing formats (when these are fully confirmed) into the , as yet unbuilt or designed, system. As part of this I’ve been looking at the Chris Gutteridge’s Grinder, a parser for getting RDF data out of Excel and CSV files. I was reminded of Grinder whilst reading his article about Linked Data at the University of Southampton in the final ever Nodalities. Whilst Grinder itself may not be of initial use, it does give me some clues about the possibilities of transforming the data.

The project also forces me to think about how the programme would run and I suspect off the command line. If this is a safe assumption, then it means that I need to get back to Perl or use Python. Much as I like PHP, I’m not sure it is a command line language. I know it can be run as one but it always make me nervous as I don’t really consider it a system administration or data munging language. In either case, Perl and Python mean another re-learning curve, especially Perl which I last use at JISCMail a couple of years ago.

A side project that I’ve been  looking at is the real-time data storage of feeds for later mining and use. I’ve been thinking of using Node.js (and actually starting something!) and Redis to run in the background. A little side something, methinks. It does mean me learning more about Node though and gives me something tangible to build. I’ve been having a little search around the Net and came across an older post by Marshall Kirkpatrick on the NTEN blog about realtime data whilst reading about event loops in Node on the Elegant Code blog. Of course, once it is stored, it must be processed to be useful but that is the next step.

After Book Hackday, I’ve written a post about beginning to develop the Open Correspondence toolkit for the Open Knowledge Foundation’s Notebook blog. I was also contacted regarding converting the TEI XML pages into JSON, which I am currently working on.

Once I’ve done some more work on it, I’ll post the code and more about it.

I’ve been working on another project which may or may not be open. It is certainly interesting but I am not sure I can say much more than that. I hope to have a blog post up soon about it but I am rather excited by it and its possibilities.

Meanwhile, the work project continues apace with some surprising outcomes for me. Following watching a video on Facebook’s architecture, I’m beginning to see certain parts very differently. I really do hope more on this but I’ve got some building to do and a bit more delving and reading that needs completion.

The current changes are:

• additional fields in the RDF endpoint.  I still need to do some major work to JSON and XML which I hope to do for the next update.

• a basic text search

• a basic set of geographic data in the collection

• better linking from the letters to the correspondent and geographical data (NB it is still incomplete)

• some mapping with Open Layers javascript.

• a Simile timeline (which is a bit slow at the moment).

Admittedly some of this is exposing work already there but hidden. However I’ve also been working on some unicode fixes to the underlying XML which is used by the project which has meant rebuilding the tables and the Xapian indexes.

Following a request on the Open Literature mailing list, I’m looking at the idea of using Python’s NLTK to create some linguistic API wrappers around the Xapian search. It strikes me that these letters can be used to create a corpus of Dickens’s language where you can explore the language used in family correspondence (to his daughters and wife), to other authors (Wilkie Collins) and to readers. That is a longer project though in terms of building the relevant indexes.

I’m also looking at the idea of clustering a collection of letters to a correspondent and seeing what happens (for some reason, the current script is looking at Wilkie Collins). There is also a set of queries that one might run against letters discusing books and the publication dates to view the distribution. I’m working on these latter questions at the moment for intended release later this week but I do foresee it being delayed a while.

I bumped into Eamonn Neylon and we went along to the Mobile Web session with Mike Jones from Bristol and the Molly project. They outlined the two main approaches (either via the various app markets or having a front end which caters for the different phones) and issues such as being sandboxed from the hardware layer at the moment. It would seems from them that you need to do both ideally, though development time doesn’t always allow. The session was slightly hijacked by the Python 2 versus 3 question and if Molly would ever use Python 3 but we gradually got back on track. The main barrier to entry would be the lack of standardisation so you need to target the platforms as well as the hardware issues.

I stayed for the Linked Data session which Chris Gutteridge took sort of control from the array of speakers. The main focus became notions of openness (as defined in the Open Knowledge Definition) and how it is perceived by the academic community bringing back up the ideas of attribution on the web. (An issue which is partly cultural.) The issue of clear licencing came up again as well but there does seems to be some clarification needed on the different models.

I did go to some of the lightning talks before lunch but they didn’t leave much of an impression this time (though Chris Gutteridge did plug his Q &D RDF Browser which I’m thinking of using). After an excellent lunch, I wandered into basecamp where I plugged in my laptop and worked on Open Correspondence whilst waiting for the session on the Linked Data API. I did spend a couple of hours work on it to fix some bugs and little things for the next version to go live (though discovered another one in places with some missing but it is not huge, just needs a couple of hours). Rufus Pollock and Jo Walsh popped by so we managed to catch up and do some hacking. Rufus suggested using Flask which I think I’ll use for some smaller projects in the future (and for some reason Backbone was mentioned but not sure how).

I went along to Chris Dollin‘s talk on his eLDA library and how the Linked Data API works. It seems like an eminently sensible solution to removing the complexity of semantic technologies from the user and to make it easier to use. It is certainly something which will be useful to get my head around completely.

The day, as suggested by Mahendra Mahey, was definitely more useful when doing something and just cracking on with it. We need more days like this as it provides a collegiate atmosphere to try new things and take a look at different technologies which might not appear on the normal radar. The friendly atmosphere was great as well. I’ll book both days off if it comes around next year.

I’ve finally done the first pass of this update which I’m hoping to make live soon once I’ve updated the controllers and re-checked the other data improvements. Whilst it is not perfect, it is a lot better than it was. I think that the next week will be spent going over the endpoints and the Pylons controllers so that the data is cleaner than at present and correctly linked.

It has been a useful exercise in that I’ve started rewriting the parser for the letters (an ongoing large job I was thinking of doing when I come to the next set of letters) and putting some of the earlier thoughts into place.

Once I’m happy with these updates, I’ll update the site which does mean rebuilding the databases and endpoints. However once it is done, it should be a lot cleaner  and I can then start looking at the correspondents and linking into other data sources like dbpedia.org. I think that the first task though might be to restart work on the clients that I had been putting together  as a basic development kit.