Archive for the ‘weeknotes’ Category

Weeknotes: Open correspondence

Sunday, September 5th, 2010

A quiet week as I’ve been having a few days off but I’ve been working on some of the tickets for Open Correspondence.

The urls have changed to /letters/view/<author>/<correspondent>/<letter id> in an attempt to make them more user friendly and also to allow the user to define smaller or larger collections by altering the url. There is also some basic content negotation to expose the data in json, xml and rdf as well as html.

I’ve been trying to use the linked data api specs to make that the site operates as a service as well. Multi-tasking really.

One of the next things to do is to develop graphical features for the correspondent pages and to sort out the RDF with rdflib. Think I’ll try to do that next week though.

Tags: ,
Posted in projects, weeknotes | No Comments »

Weeknotes: Ubuntu, messaging and Open Correspondence

Sunday, August 29th, 2010

It has been a while since the last weeknotes. I’ve finally made the move to Linux, or at least dual booting, by installing Ubuntu so I’m currently learning a little the OS and getting a development environment set up for it.

I’ve nearly finsihed the ongoing accounts project at work. The framework is up and it went through testing over the last couple of weeks. There are a few rough edges and some bugs which still need fixing but it largely seems to be there now.

I’ve also installed the first part of a messaging server written in PHP (taking ideas and concepts from JMS and Python’s Routes for service urls) which takes a message from the core CMS system and routes them to the correct service using SOA. If there’s an issue with the service then it logs it and queues the message using Redis (athough an array might be quicker, I wanted the queue decoupled from the server if it failed or had to be restarted and the memory was wiped). I need to finish up the worker to dequeue at certain points in time but it is expected that I’ll get it finished in about four days once I’m back at work.

I’ve done one or two things on the Open Correspondence site as well. I’ve tidied up the source XML and the sources XML as well to expose them so I need to update the site itself. The next thing I think we need to do is to start writing stuff to expose the underlying data and to show what you can do with the data. One of the things that I want to do is to write a function which I can put behind either Protovis or Javascript Infovis Toolkit to convert a SPARQL query into the relevant JSON and I’m thinking of using Lee Feigenbaum’s sparql.js script. Quite possibly I need to write some sort of API to the dataset to allow other queries to be run.

My friend, Simon Biles who owns Thinking Security, and I have been talking about a Knowledge Management project which is slightly aligned with some stuff I’ve been thinking about storing research pages for RSS and web pages. He’s thinking in terms of MS Office documents which means a little investigation into the various types of structured storage in Office and the ways that Office has changed to mine different types of documents. It does appear at first glance though that newer versions of Office and Open Office are similar in terms of finding the metatadata being collections of XML documents in an archive.

Tags: , , ,
Posted in Information Retrieval, projects, weeknotes | No Comments »

Weeknotes: Talks, Open Correspondence, XMPP

Sunday, July 25th, 2010

I gave a talk at the Oxford Geek Nights about Open Correspondence and letters. At some point I really ought to learn how to give talks. Anyhow Russell Davies was the main speakers and he showed how you could make physical objects from data derived from social networks. (He has a marvellously sane post about the Raoul Moat facebook page.) Anyhow its gathered some people who are interested in contributing. Now I’ve finished the book, I’ve got more time to make changes to the codebase which urgently needs it. Finishing off stuff really. Then making the real changes.

Accounts has been slightly on hold since the wages needed to be run and I didn’t see that accounts or operations would be happy with fugures potentially changing.

The main project has been setting up a notification service to set up the service layer correctly. I’ve finally got the server working so I’m just building a framework. I thought of porting parts of djabberd projects into PHP but I’m just  looking at parts of it but XMPP is certainly a useful tool in getting machines to speak to each other and to develop event driven services.

Tags: , , ,
Posted in projects, weeknotes | No Comments »

Weeknotes: documentation, prototyping and cats

Sunday, July 11th, 2010

I’ve spent most of the week either trying to persuade colleagues that rewrites are needed to existing services. I’ve also finally managed to get the initial promise of working from home so hopefully I’ll be able to get the rewrite started on the “quiet” days away from the office. (Although the cat can drive me nuts before she goes to sleep at 10am).

Still working on the accounts project which keeps unravelling a series of underlying problems. Most of them we know about but they appear in all sorts of odd places.

Assuming the world doesn’t fall on my head next time I’m in the office, I’m going to try and spend the day at home on a “Fedex” day. I’m taking the notion from an issue of Wired where they were talking about different ways of working and Atlassian mentioned “Fedex” days where you spend a day building a prototype. What I’d really like to get prototyped is the service bus / queuing system. So fingers crossed.

The impetus came from updating the disaster recovery documentation and writing the first department of the service status documentation (which I wrote after getting the last bit of debugging finished). I know that documentation is not everybody’s favourite thing but I find it useful in rethinking the system and making sure it fits together.

I’ve made time to rewrite the load function for Open Letters. I’ve got the document building the letters in XML and written a rough upload script. Next task is to rewrite the main.py script, test the XML loading and then finished tidying up the initial document.

I’m also looking forward to Textcamp so it’ll be great to get the load finished (as it normalises the function) and get on with doing a presentation for the camp.

I’m also coming to end of writing my book on children’s fantasy. Whilst not technical in an IT sense, I’m thinking of the next project on the New Weird and how to use IT to visualise influences and timelines. The one that worries me is archiving necessary web pages for the research which I need to look towards as I’m not sure whether it is technically illegal.

Tags: ,
Posted in Open Knowledge, weeknotes | No Comments »

Weeknotes: maintenance, and Dickens

Sunday, July 4th, 2010

It seems to be maintenance season again.

Still carry on with the accounts systems and doing some work to those systems for most of the week. It is a slow job but I would rather spend time getting it right rather than rush something ut and spend the next year patching it because we rushed it rather than a need changed.

The rest of the week is spent either developing some new functionality for the admin department or thinking about revamping the existing services. Most of them are fine but a lick of paint and some further optimisation to take care of unanticipated needs wouldn’t go amiss. I suspect that maintenance isn’t high on most developer’s agendas but in a moving and growing company, some systems begin to be left behind when their use either changes or the company outgrows the service. The challenge is trying to minimise user frustration whilst getting the new version out and finished. Mm, time to work on the ‘soft skills’ of people methinks.

In the meanwhile, I’ve created an XML file of letters of Dickens and now am just changing the loading script for Open Correspondence so that there is a normalised way of loading the data into the database. The fact that everything was predicated on one file was annoying me so I made the time to change it. The next thing is to dive back into TEI lite and rework the file so that it fits into an already definted schema. (I don’t see there being any point in this case in trying to create something new as it should be unnecessary.)

Posted in weeknotes | No Comments »

Weeknotes: All quiet on the accounting front

Sunday, June 27th, 2010

It’s been a week of relative frustration with priorities suddenly being shifted and the infrastructure road map looking more and more unclear.

The soap server is largely debugged and ready for more extensive testing on the server and the back end has now been rewritten to capture more data. I cannot help feeling that it will change once more services go online to scale more efficiently but right now I don’t have the expertise to do it. I’ll get there.

On a different tack, I’m back on the accounting project that I was on several months ago and making some headway in that. Its grown since I was last involved in it but nothing that a decent set of specs and roadmaps cannot solve in terms of making it manageable.

I’ve been thinking about my next book project which is on the New Weird and genre over the last 15 years and wondering how to use dbpedia’s influencedBy and influence terms in terms of showing how writers influence each other over a century. I’m tempted to put the data into a large rdf sheet and then use javascript or PHP to transform it into JSON to see if you can use the Simile timeline software usefully or if I need to find / write something more appropriate. It does have to wait for me to finish the current book.

I forgot to link to the Open Correspondence blog post on the Open Knowledge Foundation’s blog which was posted a few days ago.

Tags: ,
Posted in projects, weeknotes | No Comments »

Weeknotes: PHP, SOAP, and Open Letters

Sunday, June 20th, 2010

It has been a fairly quiet week with the boss away. I’ve managed to complete a service to upload details from spreadsheets sent via email.

I’ve also managed to complete a SOAP service in PHP to listen for status updates and just doing the final tests to it now. Once its up it can be repurposed for other companies. One of the things that I think  will come up is how to store XML files most efficiently as MySQL 5 appears to be tied to uploading files rather than just taking POST strings. I’m thinking of using something like Oracle’s BDB XML database (though the license appears to preclude our uses) or eXist but that is something to come back to much later.

I’ve been thinking about the Open Correspondence site and the best way to allow it to be extended by other people. I think that the best way forward to create an internal XML format which the load command can use and anybody can use to create their own files and databases. Its along the lines of the stuff I partially did some work on in the Open Shakespeare project.

Given the boss is away, time for finishing more things off next week. I’ve also created a Trac instance for internal purposes but I think it’ll help on that bane if developing live – documentation.

Tags: , ,
Posted in weeknotes | No Comments »

Weeknotes: Data, service buses and trac

Sunday, June 13th, 2010

I’ve succumbed and I’ve got a microslot at the next Oxford Geek Nights where I’m talking about the Open Correspondence website. I’ve downloaded the rest of the Gutenberg copies of the Dickens letters but just need an evening to make some headway with transforming them.

I spent a fair amout of this week trying to get a status update server built using SOAP and PHP which has been an ‘interesting’ task but seems to have finally got there. Having done some debugging at home on Friday, I’ve got to test the whole thing on Monday on the test server.

I’ve also  been debugging the csv uploads into the database and refactoring the code so that there is more re-use of similar objects. On top of that I started the documentation for the services and realised that I’d written most of the upload service for invoices as well. Bonus… So all I need do really is to spend a couple of days finishing  things of at work so that the first versions of the services can go out.

Whilst doing all of that though, I realised that the queueing system that I was working on was only part of a solution to get all of our services working together. Instead of just queuing, I need to start thinking more along the lines of an enterprise service bus. So that’ll keep me busy then for a couple of weeks. My notebook has various notes and doodles, much to my boss’s enjoyment who thinks its all old-fashioned.

I’ve also started putting together a Trac instance for work to see if it scales and helps with ticketing and information acorss our department’s groups. It’ll probably be sidelined for this week whilst I try and get everything put together again with regards to the data uploads.

Tags:
Posted in projects, weeknotes | No Comments »