I needed to get some county data, and possibly other sorts, put into some forms so that it can be standardised. One of the issues that I’ve been trying to do is to make sure that data is clean across various systems.

One of the issues that has come up is the connection library to the Redis server. I’ve used Nicholas FF’s phpredis library and the Rediska library. The matters that I’ve been considering are the size and complexity of each library.

I do like phpredis but as it is a C library needs compiling so that puts a potential barrier to its use in Drupal. Also the user may not have root access to compile it. As earlier posts might show, I loved Rediska but it seems overly large and complex for such a slight task. I was running some tests today on another project and it did not appear to be closing its connections properly (an issue which might be ‘QDH’able in the short term). Both libraries are complete and large which is what makes them great Swiss Army knives for Redis and PHP.

But I would like something small and light that has access to a subset of commands and is pure PHP. I suppose a long term desire might be to implement some of the pub/sub commands but I cannot think of a use at the moment.

I’ve taken ideas from Salvatore Sanfillipo’s post which uses Ruby and ported these into PHP so the code uses Redis’s sorted sets to get the relevant items. I need to complete the code to see how it fairs under AJAX in Drupal. It would be a great sidetrack to follow some of the other possibilities such as search prediction, or perhaps this might go sideways into using that to pull up homonyms or synonyms where these are stored.

But that is definitely another day and another project.

One of them is the idea of autocomplete. Nothing new there but something that can be easily overlooked when looking at the data instead. User Experience (UX) isn’t the only reason I’ve been looking at this. I’m also trying to find ways of keeping some of our data cleaner than it is and limiting options to a validated set currently seems to be the best way of doing this. To that end, as I’m still in the early stages of learning Drupal, i searched the Drupal forums and came across this comment (http://drupal.org/node/1117562#comment-4452968) which has really helped me understand this. I’ve only implemented a cursory version of this code at the moment, throwing some rough values into the array to return, but the results appear to show fruit.

My intention is to back this onto a Redis store which will hold the cached data so that the site is not putting undue strain onto the main database and that the data store can be shared with other applications on the site. I’ve got the Rediska library which I’m currently using but it has far more functions in it than I’ll ever need so I might roll my own pure PHP stripped down library. I’ll see what time I have.

This has long-term implications in terms of continuing to ensure that the data is clean and standardised. Given that one of the forthcoming projects is based on our technical documents and converting them from existing formats (when these are fully confirmed) into the , as yet unbuilt or designed, system. As part of this I’ve been looking at the Chris Gutteridge’s Grinder, a parser for getting RDF data out of Excel and CSV files. I was reminded of Grinder whilst reading his article about Linked Data at the University of Southampton in the final ever Nodalities. Whilst Grinder itself may not be of initial use, it does give me some clues about the possibilities of transforming the data.

The project also forces me to think about how the programme would run and I suspect off the command line. If this is a safe assumption, then it means that I need to get back to Perl or use Python. Much as I like PHP, I’m not sure it is a command line language. I know it can be run as one but it always make me nervous as I don’t really consider it a system administration or data munging language. In either case, Perl and Python mean another re-learning curve, especially Perl which I last use at JISCMail a couple of years ago.

A side project that I’ve been  looking at is the real-time data storage of feeds for later mining and use. I’ve been thinking of using Node.js (and actually starting something!) and Redis to run in the background. A little side something, methinks. It does mean me learning more about Node though and gives me something tangible to build. I’ve been having a little search around the Net and came across an older post by Marshall Kirkpatrick on the NTEN blog about realtime data whilst reading about event loops in Node on the Elegant Code blog. Of course, once it is stored, it must be processed to be useful but that is the next step.

I’ve just spent the evening using Node.js and Redis, making a change from usually using Redis with PHP or occasional Python. I’ve been using the latter for logging work and messaging but have been intrigued by the idea of having a system constantly pushing out annotations if added to a document between distributed editors. The idea is that these would all be saved and pushed out to users of a particular document and stored in Redis.

Node strikes me as a good way of creating an efficient back-end system in polling Redis (or perhaps using Redis’s Pub/Sub architecture to achieve a similar end). I would need to look at whether the Publish command would save the data though.

From this evening’s hack, I’ve got the polling from the server working and need to separate out the pushing into the store from the polling to retrieve all notes/comments/et al. I suppose the next thing after that is to pop in an html page or template for a front end.

I’ll do some more tomorrow but I’m making headway with Node which is cool as I’ve been trying to find something meatier to do with it before introducing it into work, even as a development project.

I’ve played around with Redis again as part of a logging system, having used it as a really basic queue system in a previous life. I may not play with it immediately but there is a lot to chew over in the post about writing scripts against Redis.

Initially I’m looking at trying to write a search script for a day’s worth of logs, so nothing major but I’m also hoping to capture counts for services and parts of a platform being used. It’ll probably have a very few defined commands to avoid bloat although it will initially be in PHP (since I’m using it for other parts of the platform). It might change into a different language (Lua seems to be the one being muttered about) if performance requires it.

It has been a while since I’ve really played with Redis but I’m glad to come back to it and become re-acquainted with it.

Our base is database is MySQL which I like a fair amount for what we do with it but all I needed do was to cache some data. The scripts write a fair amount of data to the cache and then there is one read process to read the entire list before updating the main database. At least I know that the data has some sort of security. It is not a panacea or similar cure all but it does have a place in development for certain jobs.

Best tool and all that?

I can understand why Twitter are not using Cassandra in the main service but are still using it for other projects.  For now. Systems and priorities change and perhaps it will happen in some way.

Despite its meteoric rise, NoSQL is not the answer to everything. It does have a useful place though.

However if you think about the LLEN command, you can pop as many items as you want, drop them into an array and iterate across them. I need to try this but you could feasibly call items from the middle of the array by changing the start and end points in LLEN. Normally I’d do something like  <list name> LLEN 0, -1 for all items or <list name> LLEN 0, 2 for the first two but if you change 0 to something else where you know there are 30 items but only want 5 from position 20 then you could pop in LLEN 20, 5 to achieve the result. It is not really germaine to the queueing that I’ve been looking at (for system updates where I need everything or just the first item) but could be a useful adaptation for somebody else.

The main challenge this week has been reading Excel attachments from email. PHP’s imap library  allows you to read the structure of an email but is curiously reticent in retrieving data if you have mime parts. I spent ethe best part of a day and a half getting a script to iterate over an incoming email, filter the parts so that it just explored the attachments mime type and then retrive any attachments either from a flat structure or iterating over each part before calling imap_fetchbody(). So far the fix appears to work and has allowed me to create a prototype mail service for receiving email data. It seems odd that in the era of web services that financial data is still sent by insecure methods but we must accomodate.

I’ve also been looking at PHP’s SOAP library to create a status update service which will probably utilise Service Orientated Architecture to create a stable, scalable service. Initially I created a WSDL file using the Eclipse IDE but that threw all sorts of issues and ended up using Zend’s WSDL generator tool running across the existing server. Must look into this but there might be a conflict in versions of WSDL as well as first time learning curve. I’m hoping to get the first version of the service up this week.

I suspect that this week is going to complete the commission and service status services as well as possibly doing some documentation as it is beginning to pile up.

The first half was spent frantically converting html pages into PDFs using Verypdf’s HTMLtools server product. All in all the manual is very helpful and the test server could be set up quickly. It might have helped the other end if I’d remembered to break the file up for printing but that turned out to be a 10 minute jub to put back into production. The next task is to transfer it from the test server and onto the production one but that’ll need to wait for networking to tweak it a little.

I spent some time refactoring the call recordings archive. For some reason the archiving solution that I hacked up in November decided to start failing in March after it was changed. Despite being put back to its original state it never quite got back to working as it did. I’ve been trying to tweak it ridon and off but never found the time to complete it. I finally just made the time on friday afternoon to look at it properly. I’d been thinking about item based filtering after reading the first chapter of Toby Segaran’s Programming Collective Intelligence. (On the back of this, I think I’ll be buying his Beautiful Data at some point.)  Although this is not really an intelligent programme as such, the techniques have shown some real promise in the hurried tests. Using a Redis datastore, the percentage of found recordings is way up. Fingers crossed for Monday morning when I can see what the scripts run over the weekend. I also spent some time simplifying the matching algorithm so that I didn’t have to account for so many edge cases when dealing with time.

It seems that we are approaching some sort of real-time status update systems at work. I’ve sort of been arguing for this for a while to remove the bottlenecks of having each system dependant on another one. One of our suppliers is sending us XML data so I’ve been playing with Xpath 1.0 (since Xpath 2.0 apparently isn’t directly supported by PHP but there might be a way of passing the data to Java which adds unnecessary overhead) to extract the relevant values. Anyhow the core is running but I still need to fully test it and add in security.

I’ve also been asked to design and implement a queueing system for the main internal server. I’ve run up a quick high level overview but the detail still needs to be worked on. I’m pushing it back to June so that I can slear the decks of the older projects that are still on the board.

I had a chat with Jonathan Gray, a sound guy who does far too much, about digital humanities ideas. We’ve agreed to keep closer contact with each other about the area and to encourage each other into actually doing stuff (I have half a moleskin of ideas – time for more code, less talk then).  He proposed the Bibliographica idea in January and the team wrote a blog entry for the Open Knowledge Foundation blog. It is an idea that I’m looking forward to playing with and trying to embed data from. (http://bibliographica.org/)

One of the things that I’ve been thinking about though is increasingly when we do research, we store  web pages, blog entries and so on. Whilst there is way of recording these in a footnote (http:example.org accessed on <insert data> type thing), there does not appear to be a way of building a local archive of these with the relevant metadata for later retrieval, Don’t know about anybody else but I’ve got a fair few pages dotted around my hard drive for projects and I’d like a way of storing these properly and to be able to integrate them into bibliographies or research notes. I know that there is WARC format (Library of Congress link and the WARC tools Google code project) to play with so need to make time to do that.

I had a mini-hack on the Open Correspondence project last Sunday intending to update a couple of pages and got a little more done than that. The database needs rebuilding but the purl reference (http://purl.org/letter) now points to the schema. It is so close that I can’t wait to actually start hacking the data. Time to do the last little bits like tidy up the parser, use the weaving history API to embed a timeline and start using JENA, ARC and Chris Gutteridge’s Graphite library which worked out of the box (but as yet I haven’t entirely used it for much yet).

Goals for this week are to finish the Open Correspondence bits, update the trac instance with the various ‘todo’s, write a blog post for the Open Knowledge Foundation for Open Correspondence, do some major testing this week at work on various XML exports and imports. I should just be about caught up then. With any luck…

One of the projects that I’ve been working on is the temporary storage of information. For one reason or another, one of the workers has decided to occasionally throw a fit and not do its job properly (on top of a connection that appears to fail at odd times). What I really needed was a temporary store to save the parsed information so that if something failed, we didn’t loose everything. To that end, I’ve started looking at Redis in more detail and started using the Windows build of version 1.2.1 (available on aspninja.com) with the Rediska library. At some point I’ll sit down and compile it on my laptop under Cygwin to get the latest version.

I ended up using the PEAR version of Rediska and managed to get it up and running fairly quickly. One of the things that I needed to do was to call a new instance of the list that I was creating in each method, having split the set and get methods into two workers. The speed of Redis is fantastic and the server happily runs on the test server caching the data and allowing another worker to load into a copy of the MySQL tables that it will eventually update. I found the Rediska library really easy to use and I’ll be using it for various projects at home to do some processing rather than using MySQL all the time. Simon Willison has a post which links to a tutorial on Redis that I found extremely useful and encouraging in finding more about the server in future.

I’ve been working on the RDF exports for the open letters project which are yet to go live. The main job has been making sure that the exports validate using the RDF validator and pulling in the data. A future task is to finish tidying up the data but I’m trying to get the letter html template figured out. Since Python isn’t the main language that I know use (work is entirely based on PHP), I’ve been taking a look at the Open Shakespeare code and found that RDFa work that I worked on a year ago and completely forgotten about. It would be good to get RDFa into open correspondence but I think that is a later task. Main thing is to complete the initial port. I managed to get the www.purl.org/letter forwarding to the site but need to get a schema page up and the purl correctly referring to the right page.

One of things that I’ve been trying to play with RDFlib on Windows. I built it successfully on my last laptop (Windows XP, Cygwin) but for some reason version 2.4.2 would not build on Vista, even under easy install. I’ve been trying with the version 3 (which has just been released on may 13th according to the news group) and apparently the rdfextras project has a pure Python version of the Sparql parser which was failing to build. I’ll be trying that once the current work on open correspondent as been completed to explore what we can do with the data.

Ben O’Steen talked at the Open Knowledge conference after me and one of the things he talked about was the psutils package. I’ve found it on Cygwin and downloaded it so it would be good to have fun with that one or to find accessible Windows ports for people who don’t necessarily want to download Cygwin.