Having arrived early on Saturday and met Rachel Lawson and Euan Bayliss at NAPP Pharmaceuticals (who kindly donated the building), I caught Finn and other Drupallers. The day started off with a general welcome and then into the first session, Responsive Design using Fluid Grids, presented by Tim Davison. This is an area where I sort of know the theory but have never fully explored so was keen to learn so that sites can run on (largely) any device. I can see one or two things that I might try this out on, though not Drupal related at the moment.

I stayed in the room to hear Steven Jones giving an Introduction to Module Building. I’ve been writing modules for things for a bit but it always good to get a refresher and to rethink the basics. He also managed to pull off a live coding session in writing a quick module. I was impressed about that.

After lunch, I went to the LessCSS presentation which was more on the theory than a Drupal application. The framework solution tends towards large teams and to building maintainable large CSS files, rather than small projects. However its flexibility promises a way of reducing lines of code and making it more re-usable. There was a warning that getting it wrong could create large .less files and smaller CSS files but that may be a learning curve for developers. I like the fact that you have the choice of pre-compiling the .less files and loading the pure .css file or using a PHP or Javascript compiler to compile in the fly.

Following that Steve Purkiss gave a presentation on From Flip Charts to Features and beyond! Starting out from mind maps to find out what the clients needs are and to “think first, code later” (which is the reverse of what I believe is 37Signal’s way of build first with minimal planning). This allows you to discover and use the client’s domain language and vocabulary and allows for missing elements to be found early rather than having to rebuild later. A suggestion for then beginning to build a dev site which demonstrated the states for the client with the Context, Features and Delta projects. Using Drush make and drush site install as well builds the site and allows it to move it between servers but as yet, I have not quite managed to get this to work in a current environment.

The final session was Will Hall’s Migration, Upgrading and Moving House which focussed on the use on the Migration module and the idea of unstructured and structured developers. Again this is something to be explored with the drush make stuff to ensure that some current work can be moved forward. It does look like a good choice for ensuring that content can be moved between databases. Again more goodness to try shortly.

The Sunday was given over to a BarCamp style day which was great and sweetly flexible.

I went to Károly Négyesi‘s presentation on the Relations module. Whilst I cannot see an immediate use for it, it is always good to know about other approaches to relating content rather than just using views. There was some interest from those interested in CRM systems to set up donation forms or relations inside it. It appears to be an API to make relations much easier.

The sessions on migrating sites and high performance sites was pulled together and we overfilled the room that we started with. Steve Jones of ComputerMinds led the discussion about Aegir, Pantheon, MySQL, Varnish and Apache. He announced the Pergola project to set up a set of scripts using Puppet as a base to create a recipe based loading system to set up a site in an emergency or to ensure that changing servers can be managed to and build the site in the same way taking out any uncertainty. One useful reference is on the London Drupalcon site in the Damn Quick Drupal site. There a loads of resources and options for differing servers but if a community effort can come together to educate and share knowledge, then all the better to showcase how Drupal can be set up in high performance situations on boxes of all sizes without necessarily requiring an operations team to atleast get started.

After lunch, we talked about Drupalcamp’s: The good, the bad and the ugly in order to get a better idea of how to run one. The notion of community and shared resources was mooted and I gather that there are thoughts to resolving this. However we learned a lot for the proposed Oxford Drupalcamp in terms of organising and setting up a site using COD. One of the running themes, and this is perhaps how the Barcamp style does best, was the event is more communal and less rigid. There is more flexibility to either listen or to get  a group of people together to do something. It also encourages the attendees to participate. The UK Drupal scene does appear to be fractured and in need of a place to pull information together. Perhaps this will change, I would hope so. It would be great to have one place to find UK related Drupal information easily.

Driving back to Oxford was a chance for a good conversation with Jam, the Acquia Community Affairs manager, and Finn which made me think and reconsider ways of doing projects and community affairs. Getting slightly lost didn’t help and I’ve spent most of today recovering and writing this post. I learned a fair amount and took notes so need to spend a day or two trying things out again. That, too me, is the sign of a good meeting.

(Note re:title: traditionally, Oxford and Cambridge are rivals so the title is a reference to that. I’ve lived in or near both cities and really do not particularly care about the rivalry. It is meant as more tongue in cheek.)

One of the things that I have been looking at, partially because, as Joel on Software’s article, Painless Functional Specifications – Part 1: Why Bother?, puts it,

When you force yourself to write a good, complete spec … you notice all these things and you either fix them or at least you mark them with a big red flag.

The last company that I worked for never wrote specs. Documentation, pah. Not here. Not until my last three weeks where I wrote up the documentation on thirty odd projects. In theory, great, but I was describing the existing system, rather than having a system to check my programmes against spec. Projects were late and needed patching because there was no description of what was being planned to communicate.

I am currently working on a project which I am writing documentation for. Again I have sort of started in the wrong way. The spec has not been written at the beginning and writing it up now, I can see issues with the existing systems. I can see where they are and can fix them. No spec, not entirely clear what the issues are. It also means that other team members working on systems which interface with mine or managers need to report on progress. It takes pressure off me, the developer, to be on hand to describe how the system works or to diagnose issues before they arise (well try to anyhow).

So where to now? Well. I need to do some writing and create both functional and technical specs. I will learn, and possibly fail at some point only to pick myself up again. A starting point, apart from trying, is to read the other Joel on Software articles, keep researching and writing.

I’ve been using messaging to link distributed databases together through some homegrown middle-ware and services. In many cases the messages tends to be a subscription to a published message, a simple work queue in essence.

One issue is that Rabbit does not deal with messages in Java Messaging  System (JMS) and ActiveMQ does not deal with Advanced Message Queuing Protocol (AMQP). Fortunately Rabbit does have a Streaming Text Oriented Messaging Protocol (STOMP) connector which can be used in bridging systems together in conjunction with the AMQP Client plugin. (At one point, adding plugins to Rabbit was a pain but with the 2.70 release, it is a question of running a simple command: rabbitmq-plugins enable <plugin-name>. However I fear I may be digressing.) ActiveMQ allows STOMP as one of the enabled protocols on startup if defined as a transport connector in the XML.

As an exercise (a conversation that I had during drafting this post has meant that the  RabbitMQ servers may be taken out of commission on the current project but are better for another part of the project), I have set up a RabbitMQ service listening to and serving STOMP on port 61613 (the default) and with ActiveMQ listening to and serving STOMP from port 61618. Since I am using the same box I cannot get the two services listening to the same port. I have tried on a virtual machines to get Rabbit on one instance of Ubuntu to listen to ActiveMQ on a different box serving on port 61618 but got a badaddress error – which given this afternoon’s noodling around with Rabbit and ActiveMQ makes sense given that the RabbitMQ plugin appears to be producing and consuming. I found this frustrating since I could get the PHP STOMP library which Fusesource produce to listen to the port in the initial experiments.

My original idea was to see if I could get a message sent to Rabbit from Active. Having seen the direct connection fail, I had a look at ActiveMQ’s networking section to see if I could get alter the XML configuration to include a RabbitMQ endpoint as it suggests that this should be possible. A quick test this afternoon was not exactly successful since the broker would not start if I added the networkconnection section to the default XML. The documents would suggest that it would listen but when I got the default listening to port 61613, it appears to load but the message is not passed through from RabbitMQ to ActiveMQ. When I looked at the ActiveMQ log, I saw the error Wire format negotiation timeout: peer did not send his wire format which is a fatal error. That would suggest that the broker will do this for JMS but not STOMP but I could be wrong on that.

Taking a queue from a brief discussion of JMS to JMS bridging, a STOMP server in Python bridging a STOM consumer to a database, I began writing a bridge which listens to the RabbitMQ endpoint and then publishes to a queue on ActiveMQ (or vice versa if the configuration is switched around) and the JMS to JMS bridge ActiveMQ documentation, I wrote a light weight bridge in PHP which connects the STOMP endpoints and publishes to each queue. Creating a simple consumer/producer, since I would expect any required passing between such producers to be a direct message rather than a fan-out message or topic, I managed to get the message passed across in under a second but I need to explore the amount of microseconds. The quick test that I ran suggests that there are 0.0087219 microseconds between producing a RabbitMQ message and consuming in ActiveMQ. I need to run this several times and admittedly the performance may not be desired in financial or similar high performance systems but then would that use be trying to bridge systems rather than federating like systems instead.

The bridge could also be used to transform the messages between protocols. As I had at one point, I had one system using RabbitMQ and AMQP which ActiveMQ does not support yet and one in ActiveMQ using JMS. To reduce the need for multiple consumers to take in the same message but in different messaging formats, then the bridge could be used to transform the message into the recipient protocol which could be useful if the message being sent is a standard one.

It might also allow a flexibility in having message brokers placed in different networks in that a consumer connection does not have to be maintained across the firewalls. Rather the connection is made on production so any connection does not have to have a keep-alive call being made.

I still need to run more tests for speed and to test the inevitable speed reduction in swapping protocols but this approach appears to make solve some messaging network problems using PHP. Looking at the Camel documentation, this piece of code begins to look like doing the same sort of thing in PHP but right now I am not thinking about that (though the possible uses do come to mind).

Fortunately Drupal 7 comes with a fair few options which the P2P Foundation has documented  on its wiki.

One of the things that I have been looking at is how to get ad hoc groups of individuals interacting. Organic Groups (OG7 as I use Drupal 7) is a really powerful way of building an area which can be shared by a user and to grant options to create pieces of content from blog posts, Wiki pages and forums.

One of the early issues that I had was to create a PURL for the groups and annoyingly this does not necessarily come out of the box with the module. I came across this in the excellent presentation by Amitai Burstein at Drupalcon in London, “OG7 – this time it’s on” when the person next to me asked how one created Purls in OG7 but there was no real answer. This slightly troubled me but I did not pay it much mind at the time.

The standard way would appear to be using OG7, the Spaces and Purl  modules from Drupal and some of its other requirements. Using Spaces, the developer can build features such as blogs or similar and bundle them into the area of the group namespace. The group namespace can be created in the Purl module and it creates a series of sites within your main site. Having already started building, I was not really prepared on this occasion to change and rebuild from scratch, especially as I was trying to integrate it into an existing site with its own content.

The solution that I came up with was to explore Views 3 and OG7 and to use the tokens supplied in the core when OG7 is enabled and also an experimental module, OG Token, which supplies the last piece of the puzzle.

When OG7 and OG Token are installed and a group type is set up, the group name can be used in the url to create a front page which might be along the lines of http://somehost.com/mygroup. That gives the group “mygroup” a persistent url under which we might want to have other content types, such as forums or blogs or articles, or even some homegrown content types. To associate these with the group, a replacement URL needs to be set up in the URL aliases section. Using OG Tokens allows you to use the [node:og-group-audience] token so that you could set up http://somehost.com/mygroup/articles/[node:title] for a root for all articles belonging to that particular group and the title gives a PURL for the article as well. The group audience is set up by the user when creating the article and the audience is set. The base content type must have this set before the content is created which is outside the scope of this post.

You can use views to make the URL more useful if you take the  [node:title] away and just want to show a list of all articles belonging to that group audience. (Bear in mind that more than one audience can be set for an article.) To get this working, I altered the og_content view which comes from the OG install. Using the OG Group label (the [node:og-group-audience] part of the URL), I set up the following PHP code:
$handler->argument = str_replace("-", " ", $argument);
return TRUE;
This allows me to take the group label and, with  a relationship set up with the Group GID, return all items belonging to that group. Using a second filter, I can then limit it to which ever content type that I want to show. An easy way of showing all of these is to use the Quicktabs module (set up a block for each of the different content type) and then you can display an AJAX block on the group homepage to show latest content.

So why use the group ID rather than its name? Very simple. If you can two groups called my group, the URL is mygroup and mygroup-1. However OG does not appear to store this so calling the mygroup and then trying to search on mygroup in the database could reveal content that belongs to the wrong group. Embarrassing at best but it could also expose confidential information if one or both groups are private. Munging the URL and returning the ID means that the correct content belongs to the correct group.

For my purposes, I wanted to create some simple content types which a user can utilise to create content for any group that they wish to (providing that they belong to them) and to share them without segmenting the site into subsites with similar features in each one. Using tokens has allowed me to do this and to maintain some separation of content using views. It also means that I can set up URLs which return relevant content at each section as well in the hope that I can use this to help the user discover content as well as having groups be discoverable and persistent.

In the tutorials that I’ve seen, the gallery often links back out to the original URL but what if the listing needs to link back to a node. It is not much but I did find  away of making the link work. I like the fairly simple listing of the preview, a title and associated tags. The preview and title are obvious (and the listing could be extended by adding in a description as well.) but the tags could be used for categorising the collection folksonomically.

As well as Media, Media_Youtube, Media_Vimeo, I used Taxonomy and Views 3.

The first step was to create a video content type to take an embed url from the services and put it into Drupal and allow the user to add in a title, a description and a limited number of tags to define contents by creating a form and having the form save the node. The file URL needs to be saved as a file entity since that is what Media 2 now uses. When the form is saved, it creates a node with the url. Using the theme() functions in the templates (thanks to Finn at Ecobee.coop for helping me with this bit), the video stream can be shown using your own template. If you just use the standard node template which comes out of the Drupal box, then you do not need to worry about this at all.  So that sets up the individual nodes which have videos and descriptions of them in. But how to view them?

Well that is where views comes in. I created a new view call videos/all and used the fields to select all videos at that url. I also set up a contextual filter to query the videos which were set up with tags against them to filter on the tag term and return all associated video type nodes. The default is set to show all if there are no videos with a term.

The one issue that I did find with this was if I wanted to have the video itself linked to the node, not just the title (which can be set when you call the title as a field). I had to use the content:NID field which is excluded from the display.

In the content:Video field, I set up a link which referenced the content:NID  value in the link option, had it replace the spaces with dashes and use an absolute link to provide the URL so that a preview image could be used as a link as well like Vimeo or YouTube where you have both sorts of links in the galleries.

(Another long-winded way of doing this is using the global PHP and using drupal_lookup_path() to turn the node id into a URL but you shouldn’t need to do this. )

Using these simple modules, you can create a simple Video gallery that allows you to show videos from the various web services using the Media modules. I am sure that there are other ways of doing the same thing. I am sure that I will be exploring this more as I am hoping to use Media for a variety of other projects.

The physical book is enjoying a bit of renaissance in the sense that it appers to be viewed as thing of beauty and joy. The republishing of classics in beautiful jackets and the act of booksellers becoming publishers again shows that not only are older models of publishing/bookselling returning but also that the book’s future is not solely in electronic format.

It is a good time to rethink a cherished, for some of us, medium that we love but also to hold onto its past as somethign still vital. I’m not sure that Butterworth adds much to the argument in his article but it is a reminder that such things can, and should be, mused on.

I have found myself spinning out in various directions due to existing work and personal commitments and things I am interested in. Some of these overlap in odd ways and some do not. I have largely stopped doing anyhting personal in the last few weeks for a variety of reasons but having had a couple of days off and looking at a backlog of unpublished posts on the blog, I’ve begun rethinking some of these. I have one or two things about Drupal and messaging to be published and to explore in the next few days. Admittedly these are slightly more work related but I am also interested in them.

I am also trying to do somethings for other people. I am admittedly way behind where I would like to be on these and have used the last couple of days off to have a think about how I tackle these and play ‘catch up’ to produce the work that  I would be happy showing rather than just trying to get a post out or a function finished. There are things that I can fold into work from these projects and I am determined to get on with them next. When I do, I can post about them.

As some acquaintances know, I have a notebook full of ideas which I have collected in the last few years. The intention of these has always been to build them into something useful and to explore how large scale things and ideas can be rebuilt for the individual. At this moment I have no idea how I am necessarily going to complete them but I do see the next few days and January trying to clear up things and tie off soome loose ends.

This does mean that I will have to rationalise the things that I will get involved with. Admittedly I overcommitted last year and I am trying to honour these commitments, but many of these are in areas that I am interested in. So decisions, decisions. I started doing some of the things that I do because I was deeply frustrated with a job that I was in and followed my own curiosity. It took me into new areas and langauges. I managed to build things but perhaps it came out of balance. I got frustrated because I could not build all the things that I wanted to.

I am hoping to go back to work refreshed and invigourated as well as recomitting to the current projects. I do not want to just finish them but build something with which I am happy. Some older projects may get rebooted in the scheme of things as they become relevant to work or other projects but they may also disappear. At this moment I amnot 100% sure what will happen to them but there is a rather dusty development section which needs spring cleaning. I may also move it to a secion called, tentatively, Kitchen Table, which comes from the notion of cottage working and also that fact that my desk is an old kitchen table.

I am excited but nervous. Spring cleaning, moving on is not easy and I am sure that I will see old projects and not want to get rid of them but I need to. I need to whittle away cruft and prune to renew growth and have fun again. I am also learning to sit on my hands and not to start anything new for a couple of weeks at least whilst I finish the current bits of work I have said that I would get on with.

I believe that most of these compliment each other and the problem that I am interested in should become clearer and more focussed in due course.

Onwards and forwards.

One of his tweets is that financial publishers are transforming themselves into an information platform, following another that academic publishers are joining together to create platforms. It seems that there is some confusion, if not fear, about the future of publishing.

The idea of publishing as information, that it could be a platform, which either gives away or sells the information to the end user. The idea of a book needs to broaden from the idea of the end physical object and into the data leading to it (for academic publishing) or perhaps notes or edits for fiction.

In the Guardian Review today, Kathryn Hughes has a wonderful article on the re-rise of the beautifully designed book. It seems that physical book is being rethought as a physical object which could conceivably be kept, rather than being seen as a consumable. I’m in awe of the cover of Erin Morgenstern’s Night Circus and will treasure that hardback; that it is incredibly well written is another keeping factor here, of course. I’ve been looking at the Vintage Modern Classic books and the Faber poetry hardbacks with a desire to sit in the shop and stroke the covers.

Publishing has a wonderful future ahead of it, not only a glorious past. There are moves in the industry to the new future which bode well on the fringes as well as in the forward thinking houses. Looking forward to seeing what happens. I’m trying to keep this in mind for various forthcoming projects.

A quick Google search brought me to this post by Roy on building an HTTP service using Tornado. I am fairly new to Tornado but have been looking at it for an experiment at work. However I have been using Flask for other projects such as a quick and dirty RDF browser largely based on Chris Gutteridge’s PHP browser. (At some point very soon, I need to get back to this but other projects are slightly more pressing at the moment.)

After a quick upgrade to version 0.8, I have managed to put something together a little like Roy’s script but I’m hoping to go further and add a storage layer before tidying it all up.

Unlike Roy’s script, I’ve pushed the Tesseract and file handling code outside of the server. In the long term, I’d like to split out the file handling and storage facilities from the web server which means looking at the storage. As a quick step, I’ve popped in a link to a MySQL database but a far better option would probably be a NoSQL database like CouchDB or similar. I suppose a Key / Value store like Redis could be used as well (Redisfs apparently does something like this) as a back end. I’m keeping options open.

I do have a temptation to use RabbitMQ to notify various workers that a file exists which suggests that if I’m hoping to use this as a book scanner back end (discussed at the Textcamp event in August), then I need to add in an automated set of scripts which reads a directory and deals with the file and moving, storing and scanning them. Perhaps Tornado might be a long term answer but realistically it is not needed for a test project.

Also, Tesseract will need some training as I’ve discovered this evening playing with some newspaper text and seeing some of the results. As one of the reason I began this was to store old fanzines and newspaper articles which I’ve stored for research but are now degrading, that might be a problem.

Either way, this is a way of moving ahead with the book scanner conversation and building something small to scratch some itches.

I decided to install it on Snow Leopard, having already set up PEAR and PECL. Using pecl install apc, I downloaded APC which appeared to be fine. However when I ran some scripts, I got the error:
Fatal error: Unknown: apc_fcntl_unlock failed: in Unknown on line 0

which was definitely not expected.

I found some useful instructions on the PHP Bug fixes (https://bugs.php.net/bug.php?id=59750 ). I ran pecl uninstall apc then installed the beta version and answered “no” to pthreadm utex locks and “yes” to spinlocks. You will also need to add apc.rfc1867 = 1 to the php.ini file for use with Drupal 7.

Having done this, I no longer get any errors in the status report regarding APC and when I have got my head around Organic Groups, I’m looking forward to getting into APC and improving Drupal performance.