However one of them was a lack of testing. Not full system testing but the less glamorous and more (let’s be honest) tedious unit testing. I have ventured into testing before but not on an enterprise scale. Previous companies have not seen the benefit in spending time in creating tests and running them.

In a sense I fell into it quite by accident. I was running some manual tests for another project, following a set of use cases and test plans. All went well if slowly. Having got in early, I spent some time using the Selenium IDE in Firefox using the WYSIWYG editor in an attempt to quickly put some tests together. Having run the tests, the time taken to do them were cut down significantly but more importantly, they can be run again and again in the same way. Unless altered. Then they can be run again and repeated. Yes, setting up tests takes time but in the long run appears worth it in terms of confidence.

The confidence comes from being able to repeat things and to be able to identify either code or user story issues. If new code comes in, the code can be run and regression tests performed.

That is running tests at the end though. What about developing? That’s where Test Driven Development or TDD (Wikipedia, Agile Data on TDD, ) comes in. Or where I hope and strongly believe it will.

I am not going to go into what TDD is but drawing on current experience, it will allow me to have a greater confidence in the built code. In the long run, spending some spare time getting into writing tests and then building code around them should allow me to refactor the code and remove its issues.

Moving on though, I can use it to make sure that bad code does not creep back in. Since we use PHP at work, I have been using PHPUnit as the best framework. Having dipped in and out of PHPUnit over the last couple of years, I needed a quick reminder about some of its features and how to set it up to read existing classes which is what this sitepoint article gives.

Is this where it ends? No.

The next jump to make is to get the Jenkins Continuous Integration tool to run the tests that I write. Automatically. Whilst writing the tests and then code offers confidence that the while thing is going to do what I think it is doing, I would really like to not be continually running the tests. I have installed the Jenkins tool on to Ubuntu 10.10 (though upgraded to 11.04 recently) and once the first tests are written, then I will set up a project in Jenkins to run it. That is another post for another day.

So where does this leave me? In the first place, I am playing a certain amount of catch up to build a testing skeleton. It will have holes but the coverage can be extended and I get into better practices with regards development.

More importantly, the code that I write can be released with a certain amount of confidence and built on or against with that confidence.

I’ve just been playing with Display Suite which I have heard really good things about but never got around to using. Ivan Zugec’s tutorial (part 2 here) helped me get a start but Jyve’s post on Display Suite vs Panels and his Introduction to Display Suite is a useful intro into the ins and outs of both. I have to say that even without these, I found DisplaySuite remarkably intuitive and fairly easy to set up. It has its place as does Panels.

I’m sure that I’ll be posting more on DisplaySuite in the future.

I did manage to bend somebody’s ear (sorry if I went on a bit!) and started work on a prototype for a presence system. As there were people building cool things like games and a drum/sound machine using Pusher and Twilio amongst other things, it seemed tame.  I guess enterprise systems are really not sexy but I managed to build the skeleton of a UI and how it interacts with the underlying registry which also gives me the shape of the registry.

However the conversation go me thinking about Rabbit and coalesced a while series of thoughts that I’ve been having about discoverability and registers using messaging systems.

Distributed computing is all well and good, as are distributed services in Service Oriented Architecture (SOA). However how does any system ever know what is alive? How do they know what might have been moved if the platform becomes used across businesses? If a dataset is uploaded to  a remote service, how can it be found and its download details ascertained? Trial and error? Asking the developer, architect , support or operations team?

How do we know that friends are on Facebook? The presence system which informs us if they are online or offline. Systems become similar in distributed computing. They have end points, methods and parameters? Using different transport types means that the traditional UDDI requirement for knowing the endpoint is great but it relies on the endpoint being a web service and tending towards SOAP. Most of the time this is fine but I would suggest that in the new world of message queuing and competing standards from JMS to AMQP to STOMP and onwards, the requirement for the user to know the protocol used is a must as well as the correct port to use since not every one will use the standard ones on ActiveMQ and RabbitMQ. Also the queue name is a must and what type of queue or topic requires the message.

This also affects us when we use the cloud to store data. Where is it? How can I get it as well as how large is it? Easy questions but we might also need to consider failover protocols if it is available via more than way of getting hold of it. Discovery is not about merely finding material in an ordered fashion but also helping to stumble upon data.

But first things first. Finish the presence system, taking ideas from XMPP rosters and other similar systems. The to look at exploring expanding them. I have also been continuing expanding this into the Drupal world and have something in very early stages of being a set of modules which may just work!

I am sure more questions will need to be asked and answered found in the design and building. The initial build looks like being based around RabbitMQ and Redis but it out to have some way of being portable to ActiveMQ.

More on that anon…

Reading the University Flipped article in the May 2012 edition of Wired UK got me thinking about what video technology really begins to achieve. Combined with other forms of technology, the classroom can be transformed and collaboration made easier and quicker. Which is not to say that email does not have its place as it does. Its position as the dominant method of communication has waned, especially with smart phones.

Anyhow, I have been playing with TokBox as a way of exploring video as it is a fairly new medium to use for developing.

One of the issues is the lack of documentation so some of this will be patchy as I am still new to the system.

I downloaded the OpenTok plugin using Drush. The plugin has a few requirements but it sets up a session field in a node, or nodes. It requires clean urls being enabled, the Libraries module, json-js (you will need to rename the file to json2.js and put it into the /sites/all/libraries), and JM Torres’s jquery-opentok (again it will need renaming to jquery-opentok and placed in the same library as json2.js).

A world of warning here. This plugin fails with non-clean urls. This is apparently being looked at, but there is a way of hacking around the problem. It is messy but it does work. The session variable will not be loaded or called by the plugin so you get the grey box in the screen which will not do anything. I need to find a better way of doing this but for the moment, one copy of Drupal has the opentok_field_formatter_view() adapted to call $item['session'] to re-assign the session. This may have unintended consequences and the other Drupal copy that I have with clean urls does not need this bodge. (Update: it recreates the session so you can end up with quite a few screens of yourself on one page which is great for solo Mexican waves but not as good if you are playing nicely with Tokbox’s API or you don’t multiple visions of yourself.)

In admin/config/media/opentok, put in the OpenTok key and session id which you can generate from the TokBox API page. Save this and also tick the staging server option which allows you to use the staging server rather than production.

Either create a node or add the Opentok session field to an existing node. For instance you could create a chat node which allows you to chat to any one who is online. (One of the things that I want to look at is how to combine this with presence data.) Or you could create a page to give a talk from and viewers just subscribe to the stream.

Whilst the plugin comes with a UI, it does require a fair amount of work. It is very basic but a little retheming can improve it no end. I’ve just come across a patch to add a show/hide button which I am in the process of trying out. (Update: Installing it and it does  anice job of adding some jQuery to show/hide the box on the page but it is not quite what I am looking for at the moment.) The CSS is the next task to take on and to play with concept map I have been using to map out potential user needs.

There is a long way to go but with some digging, this plugin does appear to be useful in getting set up quickly with video.

I was helping out some colleagues by testing some existing code this week. not because I do not have any thing to do but because they are woking on something large which required some help testing and bug hunting. I worked on the test plan with my manager and extended it when I was looking at other parts of the site. This was the moment when I lost my testing “cherry” as it were and fired up the selenium IDE in Firefox.

Using its simple recording and playback facilities, I was able to reconstruct our testing steps and even add to them as I saw that assumptions had been made in the original document.

The simplest way of creating tests is to fire up Selenium and to go to the beginning of the tests that you want to run. Then click the record button and it will watch you click around the site and record a series of steps. When you click record again to stop it running, you can get it to run through all the steps or just one step using the playback buttons.

If you need to add a step, you can use the really easy interface to create or delete the steps and the actions that must take before re-running the tests that must be taken. It does take some time to set up the tests but that just be weighed up against the costs of running tests manually and perhaps missing steps.

Once the tests are complete and running, then you can save the tests using the file option and then create  a folder of them to run as required. Alternatively they can be exported into various formats including jUnit and PHPUnit.

The next step will be to add unit tests into code and try to backport some tests in existing code. Set up time is outweighed by the saved time later. I suppose a future evening project will be to put this together with a Selenium server and pull code from a repository.

The other tasks has been writing documentation for recently completed projects and some ongoing and planned ones. Having written up some user stories for new ideas and also giong back over old code has made me re-evaluate some of the code and to consider it as part of something organic. It is less of a chore now and more a help to myself and colleagues so that, when the paperwork and wikis are up to date, we could in theory have a break and if something happened, one of us can dive in. Alternatively we can make out iterative process better and stronger since we are not trying to rediscover the process of something by following code until we stumble across the answer.

I did have an interesting time with HTML5 geolocation which I need to look into more deeply. Having got some code quickly working to show a position on a map, I found some dead area in Oxford whilst showing somebody else the app. I am assuming that the geolocation runs off masts and the GPS system but it does raise a caveat to the enterprise of using pure HTML5. Need to put the code  onto a page and wander around the town with my phone pne Saturday.

Using Python’s NLTK library, I managed to extract around 60 distinct locations from his novel, Bleak House. A bit of human editing has tidied this up for me but it looks useful.

Having popped this into MySQL as I am more familiar with it than PostgreSQL and returned it using Open Layers, I’ve been looking at ways of having a query which takes the user location (once given permission) and then calculates locations that around a mile away. Or within walking distance. A bit of searching around led me to this Stackoverflow question on finding distances within a certain radius from a given location. The post is also helpful in pointing me towards some of the maths so that when I have some time, I can try to understand the underlying query.

Anyhow, HTML5 to some degree has come to the rescue with its geolocation API. I know it is not a standard at the moment (the draft I saw was 28th June, 2011), but I thought it would be fun to use it to start doing some mapping with the API, courtesy of some guidance from HTML5Doctor and HTML5demos site. It also helps me to re-use some of the data collected for the Open Correspondence project and to bring locations related to Dickens together and make them useful.

What would be interesting to see is the locations surrounding a given latitude and longitude within one mile so that the user could walk to them or find out about them and query them to see if they really want to go to that location.

Open Correspondence

Development on the Open Correspondence project has been slow to stalled for a while. I have been doing bits and pieces but sitting down with Mark McGillivray of Cottage Labs and the Open Knowledge Foundation, brought some clarity. Recently the Textus project has been announced and I have been talking with the developers to put the data onto that platform. It seems to me that it is better to pool resources and to contribute where I can. There are parts of the existing project that I like and others that need more work to make me happy and it seems right now to move onto the developing platform.

Textcamp

At Textcamp last September, one of the sessions covered DIY Bookscanners (Austgate post on Textcamp). One of the actions on the Textus wiki was OCRing text. I have posted previously about playing with Tesseract and seeing this, I emailed the humanities-dev list to explore the possibilities. To this end, I have volunteered to work on the area and will write a blog post about it There is already a large amount of work that exists, so  I am perhaps not developing anything new. However it would, I think, be interesting to develop a stand-alone system that is flexible and downloadable. Like other OKF projects, it will be a Python project but also be a hardware project to try and extend some of the existing projects.

Other Bits

I’ve been working on an indexing project which appears to be coming together quite nicely. Hopefully I’ll be able to say some more shortly but it depends on a conversation that has yet to be had.

Next week, after a break, is a return to work and to data. The Dev8d conference provided me with some ideas and clarity on one or two things, so time to put them into practice.

Having arrived early on Saturday and met Rachel Lawson and Euan Bayliss at NAPP Pharmaceuticals (who kindly donated the building), I caught Finn and other Drupallers. The day started off with a general welcome and then into the first session, Responsive Design using Fluid Grids, presented by Tim Davison. This is an area where I sort of know the theory but have never fully explored so was keen to learn so that sites can run on (largely) any device. I can see one or two things that I might try this out on, though not Drupal related at the moment.

I stayed in the room to hear Steven Jones giving an Introduction to Module Building. I’ve been writing modules for things for a bit but it always good to get a refresher and to rethink the basics. He also managed to pull off a live coding session in writing a quick module. I was impressed about that.

After lunch, I went to the LessCSS presentation which was more on the theory than a Drupal application. The framework solution tends towards large teams and to building maintainable large CSS files, rather than small projects. However its flexibility promises a way of reducing lines of code and making it more re-usable. There was a warning that getting it wrong could create large .less files and smaller CSS files but that may be a learning curve for developers. I like the fact that you have the choice of pre-compiling the .less files and loading the pure .css file or using a PHP or Javascript compiler to compile in the fly.

Following that Steve Purkiss gave a presentation on From Flip Charts to Features and beyond! Starting out from mind maps to find out what the clients needs are and to “think first, code later” (which is the reverse of what I believe is 37Signal’s way of build first with minimal planning). This allows you to discover and use the client’s domain language and vocabulary and allows for missing elements to be found early rather than having to rebuild later. A suggestion for then beginning to build a dev site which demonstrated the states for the client with the Context, Features and Delta projects. Using Drush make and drush site install as well builds the site and allows it to move it between servers but as yet, I have not quite managed to get this to work in a current environment.

The final session was Will Hall’s Migration, Upgrading and Moving House which focussed on the use on the Migration module and the idea of unstructured and structured developers. Again this is something to be explored with the drush make stuff to ensure that some current work can be moved forward. It does look like a good choice for ensuring that content can be moved between databases. Again more goodness to try shortly.

The Sunday was given over to a BarCamp style day which was great and sweetly flexible.

I went to Károly Négyesi‘s presentation on the Relations module. Whilst I cannot see an immediate use for it, it is always good to know about other approaches to relating content rather than just using views. There was some interest from those interested in CRM systems to set up donation forms or relations inside it. It appears to be an API to make relations much easier.

The sessions on migrating sites and high performance sites was pulled together and we overfilled the room that we started with. Steve Jones of ComputerMinds led the discussion about Aegir, Pantheon, MySQL, Varnish and Apache. He announced the Pergola project to set up a set of scripts using Puppet as a base to create a recipe based loading system to set up a site in an emergency or to ensure that changing servers can be managed to and build the site in the same way taking out any uncertainty. One useful reference is on the London Drupalcon site in the Damn Quick Drupal site. There a loads of resources and options for differing servers but if a community effort can come together to educate and share knowledge, then all the better to showcase how Drupal can be set up in high performance situations on boxes of all sizes without necessarily requiring an operations team to atleast get started.

After lunch, we talked about Drupalcamp’s: The good, the bad and the ugly in order to get a better idea of how to run one. The notion of community and shared resources was mooted and I gather that there are thoughts to resolving this. However we learned a lot for the proposed Oxford Drupalcamp in terms of organising and setting up a site using COD. One of the running themes, and this is perhaps how the Barcamp style does best, was the event is more communal and less rigid. There is more flexibility to either listen or to get  a group of people together to do something. It also encourages the attendees to participate. The UK Drupal scene does appear to be fractured and in need of a place to pull information together. Perhaps this will change, I would hope so. It would be great to have one place to find UK related Drupal information easily.

Driving back to Oxford was a chance for a good conversation with Jam, the Acquia Community Affairs manager, and Finn which made me think and reconsider ways of doing projects and community affairs. Getting slightly lost didn’t help and I’ve spent most of today recovering and writing this post. I learned a fair amount and took notes so need to spend a day or two trying things out again. That, too me, is the sign of a good meeting.

(Note re:title: traditionally, Oxford and Cambridge are rivals so the title is a reference to that. I’ve lived in or near both cities and really do not particularly care about the rivalry. It is meant as more tongue in cheek.)

One of the things that I have been looking at, partially because, as Joel on Software’s article, Painless Functional Specifications – Part 1: Why Bother?, puts it,

When you force yourself to write a good, complete spec … you notice all these things and you either fix them or at least you mark them with a big red flag.

The last company that I worked for never wrote specs. Documentation, pah. Not here. Not until my last three weeks where I wrote up the documentation on thirty odd projects. In theory, great, but I was describing the existing system, rather than having a system to check my programmes against spec. Projects were late and needed patching because there was no description of what was being planned to communicate.

I am currently working on a project which I am writing documentation for. Again I have sort of started in the wrong way. The spec has not been written at the beginning and writing it up now, I can see issues with the existing systems. I can see where they are and can fix them. No spec, not entirely clear what the issues are. It also means that other team members working on systems which interface with mine or managers need to report on progress. It takes pressure off me, the developer, to be on hand to describe how the system works or to diagnose issues before they arise (well try to anyhow).

So where to now? Well. I need to do some writing and create both functional and technical specs. I will learn, and possibly fail at some point only to pick myself up again. A starting point, apart from trying, is to read the other Joel on Software articles, keep researching and writing.

I’ve been using messaging to link distributed databases together through some homegrown middle-ware and services. In many cases the messages tends to be a subscription to a published message, a simple work queue in essence.

One issue is that Rabbit does not deal with messages in Java Messaging  System (JMS) and ActiveMQ does not deal with Advanced Message Queuing Protocol (AMQP). Fortunately Rabbit does have a Streaming Text Oriented Messaging Protocol (STOMP) connector which can be used in bridging systems together in conjunction with the AMQP Client plugin. (At one point, adding plugins to Rabbit was a pain but with the 2.70 release, it is a question of running a simple command: rabbitmq-plugins enable <plugin-name>. However I fear I may be digressing.) ActiveMQ allows STOMP as one of the enabled protocols on startup if defined as a transport connector in the XML.

As an exercise (a conversation that I had during drafting this post has meant that the  RabbitMQ servers may be taken out of commission on the current project but are better for another part of the project), I have set up a RabbitMQ service listening to and serving STOMP on port 61613 (the default) and with ActiveMQ listening to and serving STOMP from port 61618. Since I am using the same box I cannot get the two services listening to the same port. I have tried on a virtual machines to get Rabbit on one instance of Ubuntu to listen to ActiveMQ on a different box serving on port 61618 but got a badaddress error – which given this afternoon’s noodling around with Rabbit and ActiveMQ makes sense given that the RabbitMQ plugin appears to be producing and consuming. I found this frustrating since I could get the PHP STOMP library which Fusesource produce to listen to the port in the initial experiments.

My original idea was to see if I could get a message sent to Rabbit from Active. Having seen the direct connection fail, I had a look at ActiveMQ’s networking section to see if I could get alter the XML configuration to include a RabbitMQ endpoint as it suggests that this should be possible. A quick test this afternoon was not exactly successful since the broker would not start if I added the networkconnection section to the default XML. The documents would suggest that it would listen but when I got the default listening to port 61613, it appears to load but the message is not passed through from RabbitMQ to ActiveMQ. When I looked at the ActiveMQ log, I saw the error Wire format negotiation timeout: peer did not send his wire format which is a fatal error. That would suggest that the broker will do this for JMS but not STOMP but I could be wrong on that.

Taking a queue from a brief discussion of JMS to JMS bridging, a STOMP server in Python bridging a STOM consumer to a database, I began writing a bridge which listens to the RabbitMQ endpoint and then publishes to a queue on ActiveMQ (or vice versa if the configuration is switched around) and the JMS to JMS bridge ActiveMQ documentation, I wrote a light weight bridge in PHP which connects the STOMP endpoints and publishes to each queue. Creating a simple consumer/producer, since I would expect any required passing between such producers to be a direct message rather than a fan-out message or topic, I managed to get the message passed across in under a second but I need to explore the amount of microseconds. The quick test that I ran suggests that there are 0.0087219 microseconds between producing a RabbitMQ message and consuming in ActiveMQ. I need to run this several times and admittedly the performance may not be desired in financial or similar high performance systems but then would that use be trying to bridge systems rather than federating like systems instead.

The bridge could also be used to transform the messages between protocols. As I had at one point, I had one system using RabbitMQ and AMQP which ActiveMQ does not support yet and one in ActiveMQ using JMS. To reduce the need for multiple consumers to take in the same message but in different messaging formats, then the bridge could be used to transform the message into the recipient protocol which could be useful if the message being sent is a standard one.

It might also allow a flexibility in having message brokers placed in different networks in that a consumer connection does not have to be maintained across the firewalls. Rather the connection is made on production so any connection does not have to have a keep-alive call being made.

I still need to run more tests for speed and to test the inevitable speed reduction in swapping protocols but this approach appears to make solve some messaging network problems using PHP. Looking at the Camel documentation, this piece of code begins to look like doing the same sort of thing in PHP but right now I am not thinking about that (though the possible uses do come to mind).