Posts Tagged ‘redis’

Finding a space for NoSQL

Tuesday, July 20th, 2010

ReadWriteWeb have a post on NoSQL (again?) by Audrey Watters which is a brief overview of the area.  The original post points the Heroku blog, where Adam Wiggins outlines the uses of NoSQL. I’m not an expert by any means but use Redis on a daily basis with the Rediska PHP library. I remember having an argument with the IT director when I originally proposed using Redis but I’m glad that the gamble has paid off. The caching system that uses is now far more productive than the earlier version.

Our base is database is MySQL which I like a fair amount for what we do with it but all I needed do was to cache some data. The scripts write a fair amount of data to the cache and then there is one read process to read the entire list before updating the main database. At least I know that the data has some sort of security. It is not a panacea or similar cure all but it does have a place in development for certain jobs.

Best tool and all that?

I can understand why Twitter are not using Cassandra in the main service but are still using it for other projects.  For now. Systems and priorities change and perhaps it will happen in some way.

Despite its meteoric rise, NoSQL is not the answer to everything. It does have a useful place though.

Tags: , ,
Posted in Information Retrieval | No Comments »

Weeknotes: Redis, PHP, mail and SOAP

Sunday, June 6th, 2010

I’ve spent some time writing a queueing library using Redis as a backend. I started with the notion that it would need to be a FIFO queue but didn’t want to only use the in-built parts of PHP as a stack using array_pop or array_push. Whilst it might be faster, it doesn’t allow for queue storage if the worker / router calling the queue does not run until a certain time so I looked at Redis. I  drew some inspiration from MEMQ, a queue implementation using memcached. I wrote a quick set of functions to handle connection, enqueuing and dequeueing with the ever present Rediska as the underlying Redis connection library. I’m tempted to revisit this and to write my own connection to remove the reliance on Rediska. What I did learn was how to increase and decrease the number of items that could be dequeued. For some stupid reason, I’d got into my head that it would either by one or all items.

However if you think about the LLEN command, you can pop as many items as you want, drop them into an array and iterate across them. I need to try this but you could feasibly call items from the middle of the array by changing the start and end points in LLEN. Normally I’d do something like  <list name> LLEN 0, -1 for all items or <list name> LLEN 0, 2 for the first two but if you change 0 to something else where you know there are 30 items but only want 5 from position 20 then you could pop in LLEN 20, 5 to achieve the result. It is not really germaine to the queueing that I’ve been looking at (for system updates where I need everything or just the first item) but could be a useful adaptation for somebody else.

The main challenge this week has been reading Excel attachments from email. PHP’s imap library  allows you to read the structure of an email but is curiously reticent in retrieving data if you have mime parts. I spent ethe best part of a day and a half getting a script to iterate over an incoming email, filter the parts so that it just explored the attachments mime type and then retrive any attachments either from a flat structure or iterating over each part before calling imap_fetchbody(). So far the fix appears to work and has allowed me to create a prototype mail service for receiving email data. It seems odd that in the era of web services that financial data is still sent by insecure methods but we must accomodate.

I’ve also been looking at PHP’s SOAP library to create a status update service which will probably utilise Service Orientated Architecture to create a stable, scalable service. Initially I created a WSDL file using the Eclipse IDE but that threw all sorts of issues and ended up using Zend’s WSDL generator tool running across the existing server. Must look into this but there might be a conflict in versions of WSDL as well as first time learning curve. I’m hoping to get the first version of the service up this week.

I suspect that this week is going to complete the commission and service status services as well as possibly doing some documentation as it is beginning to pile up.

Tags: , ,
Posted in Information Retrieval, projects | No Comments »

Weeknotes: Data mining, XML and bibliographies

Sunday, May 23rd, 2010

It seems to be have been a week of frantic completion and refactoring.

The first half was spent frantically converting html pages into PDFs using Verypdf’s HTMLtools server product. All in all the manual is very helpful and the test server could be set up quickly. It might have helped the other end if I’d remembered to break the file up for printing but that turned out to be a 10 minute jub to put back into production. The next task is to transfer it from the test server and onto the production one but that’ll need to wait for networking to tweak it a little.

I spent some time refactoring the call recordings archive. For some reason the archiving solution that I hacked up in November decided to start failing in March after it was changed. Despite being put back to its original state it never quite got back to working as it did. I’ve been trying to tweak it ridon and off but never found the time to complete it. I finally just made the time on friday afternoon to look at it properly. I’d been thinking about item based filtering after reading the first chapter of Toby Segaran’s Programming Collective Intelligence. (On the back of this, I think I’ll be buying his Beautiful Data at some point.)  Although this is not really an intelligent programme as such, the techniques have shown some real promise in the hurried tests. Using a Redis datastore, the percentage of found recordings is way up. Fingers crossed for Monday morning when I can see what the scripts run over the weekend. I also spent some time simplifying the matching algorithm so that I didn’t have to account for so many edge cases when dealing with time.

It seems that we are approaching some sort of real-time status update systems at work. I’ve sort of been arguing for this for a while to remove the bottlenecks of having each system dependant on another one. One of our suppliers is sending us XML data so I’ve been playing with Xpath 1.0 (since Xpath 2.0 apparently isn’t directly supported by PHP but there might be a way of passing the data to Java which adds unnecessary overhead) to extract the relevant values. Anyhow the core is running but I still need to fully test it and add in security.

I’ve also been asked to design and implement a queueing system for the main internal server. I’ve run up a quick high level overview but the detail still needs to be worked on. I’m pushing it back to June so that I can slear the decks of the older projects that are still on the board.

I had a chat with Jonathan Gray, a sound guy who does far too much, about digital humanities ideas. We’ve agreed to keep closer contact with each other about the area and to encourage each other into actually doing stuff (I have half a moleskin of ideas – time for more code, less talk then).  He proposed the Bibliographica idea in January and the team wrote a blog entry for the Open Knowledge Foundation blog. It is an idea that I’m looking forward to playing with and trying to embed data from. (http://bibliographica.org/)

One of the things that I’ve been thinking about though is increasingly when we do research, we store  web pages, blog entries and so on. Whilst there is way of recording these in a footnote (http:example.org accessed on <insert data> type thing), there does not appear to be a way of building a local archive of these with the relevant metadata for later retrieval, Don’t know about anybody else but I’ve got a fair few pages dotted around my hard drive for projects and I’d like a way of storing these properly and to be able to integrate them into bibliographies or research notes. I know that there is WARC format (Library of Congress link and the WARC tools Google code project) to play with so need to make time to do that.

I had a mini-hack on the Open Correspondence project last Sunday intending to update a couple of pages and got a little more done than that. The database needs rebuilding but the purl reference (http://purl.org/letter) now points to the schema. It is so close that I can’t wait to actually start hacking the data. Time to do the last little bits like tidy up the parser, use the weaving history API to embed a timeline and start using JENA, ARC and Chris Gutteridge’s Graphite library which worked out of the box (but as yet I haven’t entirely used it for much yet).

Goals for this week are to finish the Open Correspondence bits, update the trac instance with the various ‘todo’s, write a blog post for the Open Knowledge Foundation for Open Correspondence, do some major testing this week at work on various XML exports and imports. I should just be about caught up then. With any luck…

Tags: , , ,
Posted in Information Retrieval, Open Knowledge, projects | No Comments »

Weeknotes: Redis, RDF, rdflib and openletters

Saturday, May 15th, 2010

I’ve been trying to play catch up this week at work.

One of the projects that I’ve been working on is the temporary storage of information. For one reason or another, one of the workers has decided to occasionally throw a fit and not do its job properly (on top of a connection that appears to fail at odd times). What I really needed was a temporary store to save the parsed information so that if something failed, we didn’t loose everything. To that end, I’ve started looking at Redis in more detail and started using the Windows build of version 1.2.1 (available on aspninja.com) with the Rediska library. At some point I’ll sit down and compile it on my laptop under Cygwin to get the latest version.

I ended up using the PEAR version of Rediska and managed to get it up and running fairly quickly. One of the things that I needed to do was to call a new instance of the list that I was creating in each method, having split the set and get methods into two workers. The speed of Redis is fantastic and the server happily runs on the test server caching the data and allowing another worker to load into a copy of the MySQL tables that it will eventually update. I found the Rediska library really easy to use and I’ll be using it for various projects at home to do some processing rather than using MySQL all the time. Simon Willison has a post which links to a tutorial on Redis that I found extremely useful and encouraging in finding more about the server in future.

I’ve been working on the RDF exports for the open letters project which are yet to go live. The main job has been making sure that the exports validate using the RDF validator and pulling in the data. A future task is to finish tidying up the data but I’m trying to get the letter html template figured out. Since Python isn’t the main language that I know use (work is entirely based on PHP), I’ve been taking a look at the Open Shakespeare code and found that RDFa work that I worked on a year ago and completely forgotten about. It would be good to get RDFa into open correspondence but I think that is a later task. Main thing is to complete the initial port. I managed to get the www.purl.org/letter forwarding to the site but need to get a schema page up and the purl correctly referring to the right page.

One of things that I’ve been trying to play with RDFlib on Windows. I built it successfully on my last laptop (Windows XP, Cygwin) but for some reason version 2.4.2 would not build on Vista, even under easy install. I’ve been trying with the version 3 (which has just been released on may 13th according to the news group) and apparently the rdfextras project has a pure Python version of the Sparql parser which was failing to build. I’ll be trying that once the current work on open correspondent as been completed to explore what we can do with the data.

Ben O’Steen talked at the Open Knowledge conference after me and one of the things he talked about was the psutils package. I’ve found it on Cygwin and downloaded it so it would be good to have fun with that one or to find accessible Windows ports for people who don’t necessarily want to download Cygwin.

Tags: ,
Posted in Open Knowledge, projects | No Comments »