I did manage to bend somebody’s ear (sorry if I went on a bit!) and started work on a prototype for a presence system. As there were people building cool things like games and a drum/sound machine using Pusher and Twilio amongst other things, it seemed tame.  I guess enterprise systems are really not sexy but I managed to build the skeleton of a UI and how it interacts with the underlying registry which also gives me the shape of the registry.

However the conversation go me thinking about Rabbit and coalesced a while series of thoughts that I’ve been having about discoverability and registers using messaging systems.

Distributed computing is all well and good, as are distributed services in Service Oriented Architecture (SOA). However how does any system ever know what is alive? How do they know what might have been moved if the platform becomes used across businesses? If a dataset is uploaded to  a remote service, how can it be found and its download details ascertained? Trial and error? Asking the developer, architect , support or operations team?

How do we know that friends are on Facebook? The presence system which informs us if they are online or offline. Systems become similar in distributed computing. They have end points, methods and parameters? Using different transport types means that the traditional UDDI requirement for knowing the endpoint is great but it relies on the endpoint being a web service and tending towards SOAP. Most of the time this is fine but I would suggest that in the new world of message queuing and competing standards from JMS to AMQP to STOMP and onwards, the requirement for the user to know the protocol used is a must as well as the correct port to use since not every one will use the standard ones on ActiveMQ and RabbitMQ. Also the queue name is a must and what type of queue or topic requires the message.

This also affects us when we use the cloud to store data. Where is it? How can I get it as well as how large is it? Easy questions but we might also need to consider failover protocols if it is available via more than way of getting hold of it. Discovery is not about merely finding material in an ordered fashion but also helping to stumble upon data.

But first things first. Finish the presence system, taking ideas from XMPP rosters and other similar systems. The to look at exploring expanding them. I have also been continuing expanding this into the Drupal world and have something in very early stages of being a set of modules which may just work!

I am sure more questions will need to be asked and answered found in the design and building. The initial build looks like being based around RabbitMQ and Redis but it out to have some way of being portable to ActiveMQ.

More on that anon…

Reading the University Flipped article in the May 2012 edition of Wired UK got me thinking about what video technology really begins to achieve. Combined with other forms of technology, the classroom can be transformed and collaboration made easier and quicker. Which is not to say that email does not have its place as it does. Its position as the dominant method of communication has waned, especially with smart phones.

Anyhow, I have been playing with TokBox as a way of exploring video as it is a fairly new medium to use for developing.

One of the issues is the lack of documentation so some of this will be patchy as I am still new to the system.

I downloaded the OpenTok plugin using Drush. The plugin has a few requirements but it sets up a session field in a node, or nodes. It requires clean urls being enabled, the Libraries module, json-js (you will need to rename the file to json2.js and put it into the /sites/all/libraries), and JM Torres’s jquery-opentok (again it will need renaming to jquery-opentok and placed in the same library as json2.js).

A world of warning here. This plugin fails with non-clean urls. This is apparently being looked at, but there is a way of hacking around the problem. It is messy but it does work. The session variable will not be loaded or called by the plugin so you get the grey box in the screen which will not do anything. I need to find a better way of doing this but for the moment, one copy of Drupal has the opentok_field_formatter_view() adapted to call $item['session'] to re-assign the session. This may have unintended consequences and the other Drupal copy that I have with clean urls does not need this bodge. (Update: it recreates the session so you can end up with quite a few screens of yourself on one page which is great for solo Mexican waves but not as good if you are playing nicely with Tokbox’s API or you don’t multiple visions of yourself.)

In admin/config/media/opentok, put in the OpenTok key and session id which you can generate from the TokBox API page. Save this and also tick the staging server option which allows you to use the staging server rather than production.

Either create a node or add the Opentok session field to an existing node. For instance you could create a chat node which allows you to chat to any one who is online. (One of the things that I want to look at is how to combine this with presence data.) Or you could create a page to give a talk from and viewers just subscribe to the stream.

Whilst the plugin comes with a UI, it does require a fair amount of work. It is very basic but a little retheming can improve it no end. I’ve just come across a patch to add a show/hide button which I am in the process of trying out. (Update: Installing it and it does  anice job of adding some jQuery to show/hide the box on the page but it is not quite what I am looking for at the moment.) The CSS is the next task to take on and to play with concept map I have been using to map out potential user needs.

There is a long way to go but with some digging, this plugin does appear to be useful in getting set up quickly with video.

Using Python’s NLTK library, I managed to extract around 60 distinct locations from his novel, Bleak House. A bit of human editing has tidied this up for me but it looks useful.

Having popped this into MySQL as I am more familiar with it than PostgreSQL and returned it using Open Layers, I’ve been looking at ways of having a query which takes the user location (once given permission) and then calculates locations that around a mile away. Or within walking distance. A bit of searching around led me to this Stackoverflow question on finding distances within a certain radius from a given location. The post is also helpful in pointing me towards some of the maths so that when I have some time, I can try to understand the underlying query.

Anyhow, HTML5 to some degree has come to the rescue with its geolocation API. I know it is not a standard at the moment (the draft I saw was 28th June, 2011), but I thought it would be fun to use it to start doing some mapping with the API, courtesy of some guidance from HTML5Doctor and HTML5demos site. It also helps me to re-use some of the data collected for the Open Correspondence project and to bring locations related to Dickens together and make them useful.

What would be interesting to see is the locations surrounding a given latitude and longitude within one mile so that the user could walk to them or find out about them and query them to see if they really want to go to that location.

Open Correspondence

Development on the Open Correspondence project has been slow to stalled for a while. I have been doing bits and pieces but sitting down with Mark McGillivray of Cottage Labs and the Open Knowledge Foundation, brought some clarity. Recently the Textus project has been announced and I have been talking with the developers to put the data onto that platform. It seems to me that it is better to pool resources and to contribute where I can. There are parts of the existing project that I like and others that need more work to make me happy and it seems right now to move onto the developing platform.

Textcamp

At Textcamp last September, one of the sessions covered DIY Bookscanners (Austgate post on Textcamp). One of the actions on the Textus wiki was OCRing text. I have posted previously about playing with Tesseract and seeing this, I emailed the humanities-dev list to explore the possibilities. To this end, I have volunteered to work on the area and will write a blog post about it There is already a large amount of work that exists, so  I am perhaps not developing anything new. However it would, I think, be interesting to develop a stand-alone system that is flexible and downloadable. Like other OKF projects, it will be a Python project but also be a hardware project to try and extend some of the existing projects.

Other Bits

I’ve been working on an indexing project which appears to be coming together quite nicely. Hopefully I’ll be able to say some more shortly but it depends on a conversation that has yet to be had.

Next week, after a break, is a return to work and to data. The Dev8d conference provided me with some ideas and clarity on one or two things, so time to put them into practice.

One of the things that I have been looking at, partially because, as Joel on Software’s article, Painless Functional Specifications – Part 1: Why Bother?, puts it,

When you force yourself to write a good, complete spec … you notice all these things and you either fix them or at least you mark them with a big red flag.

The last company that I worked for never wrote specs. Documentation, pah. Not here. Not until my last three weeks where I wrote up the documentation on thirty odd projects. In theory, great, but I was describing the existing system, rather than having a system to check my programmes against spec. Projects were late and needed patching because there was no description of what was being planned to communicate.

I am currently working on a project which I am writing documentation for. Again I have sort of started in the wrong way. The spec has not been written at the beginning and writing it up now, I can see issues with the existing systems. I can see where they are and can fix them. No spec, not entirely clear what the issues are. It also means that other team members working on systems which interface with mine or managers need to report on progress. It takes pressure off me, the developer, to be on hand to describe how the system works or to diagnose issues before they arise (well try to anyhow).

So where to now? Well. I need to do some writing and create both functional and technical specs. I will learn, and possibly fail at some point only to pick myself up again. A starting point, apart from trying, is to read the other Joel on Software articles, keep researching and writing.

In the tutorials that I’ve seen, the gallery often links back out to the original URL but what if the listing needs to link back to a node. It is not much but I did find  away of making the link work. I like the fairly simple listing of the preview, a title and associated tags. The preview and title are obvious (and the listing could be extended by adding in a description as well.) but the tags could be used for categorising the collection folksonomically.

As well as Media, Media_Youtube, Media_Vimeo, I used Taxonomy and Views 3.

The first step was to create a video content type to take an embed url from the services and put it into Drupal and allow the user to add in a title, a description and a limited number of tags to define contents by creating a form and having the form save the node. The file URL needs to be saved as a file entity since that is what Media 2 now uses. When the form is saved, it creates a node with the url. Using the theme() functions in the templates (thanks to Finn at Ecobee.coop for helping me with this bit), the video stream can be shown using your own template. If you just use the standard node template which comes out of the Drupal box, then you do not need to worry about this at all.  So that sets up the individual nodes which have videos and descriptions of them in. But how to view them?

Well that is where views comes in. I created a new view call videos/all and used the fields to select all videos at that url. I also set up a contextual filter to query the videos which were set up with tags against them to filter on the tag term and return all associated video type nodes. The default is set to show all if there are no videos with a term.

The one issue that I did find with this was if I wanted to have the video itself linked to the node, not just the title (which can be set when you call the title as a field). I had to use the content:NID field which is excluded from the display.

In the content:Video field, I set up a link which referenced the content:NID  value in the link option, had it replace the spaces with dashes and use an absolute link to provide the URL so that a preview image could be used as a link as well like Vimeo or YouTube where you have both sorts of links in the galleries.

(Another long-winded way of doing this is using the global PHP and using drupal_lookup_path() to turn the node id into a URL but you shouldn’t need to do this. )

Using these simple modules, you can create a simple Video gallery that allows you to show videos from the various web services using the Media modules. I am sure that there are other ways of doing the same thing. I am sure that I will be exploring this more as I am hoping to use Media for a variety of other projects.

I have found myself spinning out in various directions due to existing work and personal commitments and things I am interested in. Some of these overlap in odd ways and some do not. I have largely stopped doing anyhting personal in the last few weeks for a variety of reasons but having had a couple of days off and looking at a backlog of unpublished posts on the blog, I’ve begun rethinking some of these. I have one or two things about Drupal and messaging to be published and to explore in the next few days. Admittedly these are slightly more work related but I am also interested in them.

I am also trying to do somethings for other people. I am admittedly way behind where I would like to be on these and have used the last couple of days off to have a think about how I tackle these and play ‘catch up’ to produce the work that  I would be happy showing rather than just trying to get a post out or a function finished. There are things that I can fold into work from these projects and I am determined to get on with them next. When I do, I can post about them.

As some acquaintances know, I have a notebook full of ideas which I have collected in the last few years. The intention of these has always been to build them into something useful and to explore how large scale things and ideas can be rebuilt for the individual. At this moment I have no idea how I am necessarily going to complete them but I do see the next few days and January trying to clear up things and tie off soome loose ends.

This does mean that I will have to rationalise the things that I will get involved with. Admittedly I overcommitted last year and I am trying to honour these commitments, but many of these are in areas that I am interested in. So decisions, decisions. I started doing some of the things that I do because I was deeply frustrated with a job that I was in and followed my own curiosity. It took me into new areas and langauges. I managed to build things but perhaps it came out of balance. I got frustrated because I could not build all the things that I wanted to.

I am hoping to go back to work refreshed and invigourated as well as recomitting to the current projects. I do not want to just finish them but build something with which I am happy. Some older projects may get rebooted in the scheme of things as they become relevant to work or other projects but they may also disappear. At this moment I amnot 100% sure what will happen to them but there is a rather dusty development section which needs spring cleaning. I may also move it to a secion called, tentatively, Kitchen Table, which comes from the notion of cottage working and also that fact that my desk is an old kitchen table.

I am excited but nervous. Spring cleaning, moving on is not easy and I am sure that I will see old projects and not want to get rid of them but I need to. I need to whittle away cruft and prune to renew growth and have fun again. I am also learning to sit on my hands and not to start anything new for a couple of weeks at least whilst I finish the current bits of work I have said that I would get on with.

I believe that most of these compliment each other and the problem that I am interested in should become clearer and more focussed in due course.

Onwards and forwards.

A quick Google search brought me to this post by Roy on building an HTTP service using Tornado. I am fairly new to Tornado but have been looking at it for an experiment at work. However I have been using Flask for other projects such as a quick and dirty RDF browser largely based on Chris Gutteridge’s PHP browser. (At some point very soon, I need to get back to this but other projects are slightly more pressing at the moment.)

After a quick upgrade to version 0.8, I have managed to put something together a little like Roy’s script but I’m hoping to go further and add a storage layer before tidying it all up.

Unlike Roy’s script, I’ve pushed the Tesseract and file handling code outside of the server. In the long term, I’d like to split out the file handling and storage facilities from the web server which means looking at the storage. As a quick step, I’ve popped in a link to a MySQL database but a far better option would probably be a NoSQL database like CouchDB or similar. I suppose a Key / Value store like Redis could be used as well (Redisfs apparently does something like this) as a back end. I’m keeping options open.

I do have a temptation to use RabbitMQ to notify various workers that a file exists which suggests that if I’m hoping to use this as a book scanner back end (discussed at the Textcamp event in August), then I need to add in an automated set of scripts which reads a directory and deals with the file and moving, storing and scanning them. Perhaps Tornado might be a long term answer but realistically it is not needed for a test project.

Also, Tesseract will need some training as I’ve discovered this evening playing with some newspaper text and seeing some of the results. As one of the reason I began this was to store old fanzines and newspaper articles which I’ve stored for research but are now degrading, that might be a problem.

Either way, this is a way of moving ahead with the book scanner conversation and building something small to scratch some itches.

One of the matters that did concern us was the word “data” and what this might mean to literature and humanities. One assumption that we had was that it perhaps had a more defined meaning to scientists. But what is data to humanities? Is it the manuscript, the notes, or the published work? We decided that ‘Work’ might be a better word for the overarching principle.

One of the issues that is important is re-use and subsequently closing the re-used work down and making it non-open. The major party that we had in mind was Google Books. Whilst they are making good and admirable strides in the digitising of out of print works but there is no API or metadata store that can be used to mix up the data or to mine it in any other way. Effectively we end up where we started: with a technically open text tied up in ways that cannot be re-used.

Re-use and re-mix are extremely important within digital humanities. Influence and building on works are central to movements like Modernism and also ensuring that works and authors are accessible. Works are adapted and take on their own lives or segue from such moments.

The final major point was that citations and the underlying cited text should be open. Whilst the core of the principles are about the work and ensuring that it can be worked on, a fair amount of work goes into notes and annotations to the text (such as the great Annotate It tool) and these provide a meta work for people to build on. It is vital for debate that these are not put into a closed arena, not just for the sharing of notes but also building on the notes. They might also be put together into a new work or an annotated version of a work put together to build upon the work with communal notes.

This does represent a step forward in open literature and digital humanities. I really hope that debate does start and that these can be developed and make concrete.

The ever useful Elegant Code blog has a quick guide to streaming files using Node that pretty much comes out to the box, or Github. Using some of the code and upgrading my own copy of Node.js to version 0.59 and reinstalling NPM, I started streaming a couple of MP3s off my laptop as a quick experiment towards something that might be of use to me. One of the ideas that I’ve been having is creating a streaming layer using Node to reduce Apache traffic.

This is the first stirrings towards a hack that I’ve really had a chance to play with recently but I’m hoping to get back into some real Node development again. It looks like there is at least one really useful FFMPEG library to do transcoding and produce thumbnails.