Accounts has been slightly on hold since the wages needed to be run and I didn’t see that accounts or operations would be happy with fugures potentially changing.

The main project has been setting up a notification service to set up the service layer correctly. I’ve finally got the server working so I’m just building a framework. I thought of porting parts of djabberd projects into PHP but I’m just  looking at parts of it but XMPP is certainly a useful tool in getting machines to speak to each other and to develop event driven services.

Our base is database is MySQL which I like a fair amount for what we do with it but all I needed do was to cache some data. The scripts write a fair amount of data to the cache and then there is one read process to read the entire list before updating the main database. At least I know that the data has some sort of security. It is not a panacea or similar cure all but it does have a place in development for certain jobs.

Best tool and all that?

I can understand why Twitter are not using Cassandra in the main service but are still using it for other projects.  For now. Systems and priorities change and perhaps it will happen in some way.

Despite its meteoric rise, NoSQL is not the answer to everything. It does have a useful place though.

I’ve still got a fair amount of reading to do but this is the sort of project that makes me rethink the Open Letters project and how it could be used by other sites. It has also given me food for thought for work as well.

Still working on the accounts project which keeps unravelling a series of underlying problems. Most of them we know about but they appear in all sorts of odd places.

Assuming the world doesn’t fall on my head next time I’m in the office, I’m going to try and spend the day at home on a “Fedex” day. I’m taking the notion from an issue of Wired where they were talking about different ways of working and Atlassian mentioned “Fedex” days where you spend a day building a prototype. What I’d really like to get prototyped is the service bus / queuing system. So fingers crossed.

The impetus came from updating the disaster recovery documentation and writing the first department of the service status documentation (which I wrote after getting the last bit of debugging finished). I know that documentation is not everybody’s favourite thing but I find it useful in rethinking the system and making sure it fits together.

I’ve made time to rewrite the load function for Open Letters. I’ve got the document building the letters in XML and written a rough upload script. Next task is to rewrite the main.py script, test the XML loading and then finished tidying up the initial document.

I’m also looking forward to Textcamp so it’ll be great to get the load finished (as it normalises the function) and get on with doing a presentation for the camp.

I’m also coming to end of writing my book on children’s fantasy. Whilst not technical in an IT sense, I’m thinking of the next project on the New Weird and how to use IT to visualise influences and timelines. The one that worries me is archiving necessary web pages for the research which I need to look towards as I’m not sure whether it is technically illegal.

Still carry on with the accounts systems and doing some work to those systems for most of the week. It is a slow job but I would rather spend time getting it right rather than rush something ut and spend the next year patching it because we rushed it rather than a need changed.

The rest of the week is spent either developing some new functionality for the admin department or thinking about revamping the existing services. Most of them are fine but a lick of paint and some further optimisation to take care of unanticipated needs wouldn’t go amiss. I suspect that maintenance isn’t high on most developer’s agendas but in a moving and growing company, some systems begin to be left behind when their use either changes or the company outgrows the service. The challenge is trying to minimise user frustration whilst getting the new version out and finished. Mm, time to work on the ‘soft skills’ of people methinks.

In the meanwhile, I’ve created an XML file of letters of Dickens and now am just changing the loading script for Open Correspondence so that there is a normalised way of loading the data into the database. The fact that everything was predicated on one file was annoying me so I made the time to change it. The next thing is to dive back into TEI lite and rework the file so that it fits into an already definted schema. (I don’t see there being any point in this case in trying to create something new as it should be unnecessary.)

The soap server is largely debugged and ready for more extensive testing on the server and the back end has now been rewritten to capture more data. I cannot help feeling that it will change once more services go online to scale more efficiently but right now I don’t have the expertise to do it. I’ll get there.

On a different tack, I’m back on the accounting project that I was on several months ago and making some headway in that. Its grown since I was last involved in it but nothing that a decent set of specs and roadmaps cannot solve in terms of making it manageable.

I’ve been thinking about my next book project which is on the New Weird and genre over the last 15 years and wondering how to use dbpedia’s influencedBy and influence terms in terms of showing how writers influence each other over a century. I’m tempted to put the data into a large rdf sheet and then use javascript or PHP to transform it into JSON to see if you can use the Simile timeline software usefully or if I need to find / write something more appropriate. It does have to wait for me to finish the current book.

I forgot to link to the Open Correspondence blog post on the Open Knowledge Foundation’s blog which was posted a few days ago.

I’ve also managed to complete a SOAP service in PHP to listen for status updates and just doing the final tests to it now. Once its up it can be repurposed for other companies. One of the things that I think  will come up is how to store XML files most efficiently as MySQL 5 appears to be tied to uploading files rather than just taking POST strings. I’m thinking of using something like Oracle’s BDB XML database (though the license appears to preclude our uses) or eXist but that is something to come back to much later.

I’ve been thinking about the Open Correspondence site and the best way to allow it to be extended by other people. I think that the best way forward to create an internal XML format which the load command can use and anybody can use to create their own files and databases. Its along the lines of the stuff I partially did some work on in the Open Shakespeare project.

Given the boss is away, time for finishing more things off next week. I’ve also created a Trac instance for internal purposes but I think it’ll help on that bane if developing live – documentation.

I spent a fair amout of this week trying to get a status update server built using SOAP and PHP which has been an ‘interesting’ task but seems to have finally got there. Having done some debugging at home on Friday, I’ve got to test the whole thing on Monday on the test server.

I’ve also  been debugging the csv uploads into the database and refactoring the code so that there is more re-use of similar objects. On top of that I started the documentation for the services and realised that I’d written most of the upload service for invoices as well. Bonus… So all I need do really is to spend a couple of days finishing  things of at work so that the first versions of the services can go out.

Whilst doing all of that though, I realised that the queueing system that I was working on was only part of a solution to get all of our services working together. Instead of just queuing, I need to start thinking more along the lines of an enterprise service bus. So that’ll keep me busy then for a couple of weeks. My notebook has various notes and doodles, much to my boss’s enjoyment who thinks its all old-fashioned.

I’ve also started putting together a Trac instance for work to see if it scales and helps with ticketing and information acorss our department’s groups. It’ll probably be sidelined for this week whilst I try and get everything put together again with regards to the data uploads.

However if you think about the LLEN command, you can pop as many items as you want, drop them into an array and iterate across them. I need to try this but you could feasibly call items from the middle of the array by changing the start and end points in LLEN. Normally I’d do something like  <list name> LLEN 0, -1 for all items or <list name> LLEN 0, 2 for the first two but if you change 0 to something else where you know there are 30 items but only want 5 from position 20 then you could pop in LLEN 20, 5 to achieve the result. It is not really germaine to the queueing that I’ve been looking at (for system updates where I need everything or just the first item) but could be a useful adaptation for somebody else.

The main challenge this week has been reading Excel attachments from email. PHP’s imap library  allows you to read the structure of an email but is curiously reticent in retrieving data if you have mime parts. I spent ethe best part of a day and a half getting a script to iterate over an incoming email, filter the parts so that it just explored the attachments mime type and then retrive any attachments either from a flat structure or iterating over each part before calling imap_fetchbody(). So far the fix appears to work and has allowed me to create a prototype mail service for receiving email data. It seems odd that in the era of web services that financial data is still sent by insecure methods but we must accomodate.

I’ve also been looking at PHP’s SOAP library to create a status update service which will probably utilise Service Orientated Architecture to create a stable, scalable service. Initially I created a WSDL file using the Eclipse IDE but that threw all sorts of issues and ended up using Zend’s WSDL generator tool running across the existing server. Must look into this but there might be a conflict in versions of WSDL as well as first time learning curve. I’m hoping to get the first version of the service up this week.

I suspect that this week is going to complete the commission and service status services as well as possibly doing some documentation as it is beginning to pile up.

One of the next things to do is to reformat the dates into human readable format. I had thought of using Python’s datetime strftime to reformat the date from its current ISO format (YYYY-MM-DD) into day, month year. Unfortunately, the method states ” years before 1900 cannot be used.” A slight cramp in the plan. However there is an Activestate recipe by Andrew Dalke which might do the trick or at least point me in the right direction. It is one of the things to be tidied up at some point.

It is a good feeling to have the site running now. The next task is to write the tests and then  to refactor the code. It is very PHPish and needs to be made more Pythonic. I’ve got an idea for trying to create a dendrogram around the textReferred element and to discover the letters and correspondents around the books that Dickens was writing. One of the tings is to continue loading the other volumes of Dickens’s letters into the site. So version 0.2 is a little way off but the light at the end of the tunnel is not a train this time.

Workwise has been a little hectic. I must make some time to write a method to allow our admin team to resubmit applications. Like so many things it is a balance between a five minute job and the two hour ones that need to be done. The major job for the week though was getting the automated printing working.

One of the jobs that admin do is to go through each client and create the packs for them. Using HTMLtools, I’ve managed to compile the html into PDF and then convert the PDF into a PostScript file for a printer. I’ve managed to use the Line Printer Remote protocol to send the job to the printer. It is a simple enough command:

lpr -S <ip address/name of printer>  -P <name of print job> (-o <optional -o 1 sets file to binary>) <name of file>

Windows doesn’t appear to support the full protocol but enough to be useful. The -o switch appears to only define whether the file is binary or not rather than specifying the paper type and so on. Annoying but it can be got around.

Anyhow it got me thinking about other ways of using commands to explore how texts can be converted and changed into useful objects. It brings me back to the use of psbook for printing but how to make it useful for an average user who does not necessarily want to run various commands. Having had a conversation with my friend Darren Nash ,editorial director of Orbit books,  about the future of publishing; he opined that small presses would come to the fore. I think, certainly in genre that this is correct. It would be interesting to see how existing tools could be used towards these ends rather than constantly re-invent the wheel.

Now that the first version of letters is out the way, time to go over other projects. I’ve got a yen to try and create something from Milton’s Areopagitica, appropriate I think as it is a cry for free presses.