One of the things that I have been looking at, partially because, as Joel on Software’s article, Painless Functional Specifications – Part 1: Why Bother?, puts it,

When you force yourself to write a good, complete spec … you notice all these things and you either fix them or at least you mark them with a big red flag.

The last company that I worked for never wrote specs. Documentation, pah. Not here. Not until my last three weeks where I wrote up the documentation on thirty odd projects. In theory, great, but I was describing the existing system, rather than having a system to check my programmes against spec. Projects were late and needed patching because there was no description of what was being planned to communicate.

I am currently working on a project which I am writing documentation for. Again I have sort of started in the wrong way. The spec has not been written at the beginning and writing it up now, I can see issues with the existing systems. I can see where they are and can fix them. No spec, not entirely clear what the issues are. It also means that other team members working on systems which interface with mine or managers need to report on progress. It takes pressure off me, the developer, to be on hand to describe how the system works or to diagnose issues before they arise (well try to anyhow).

So where to now? Well. I need to do some writing and create both functional and technical specs. I will learn, and possibly fail at some point only to pick myself up again. A starting point, apart from trying, is to read the other Joel on Software articles, keep researching and writing.

In the tutorials that I’ve seen, the gallery often links back out to the original URL but what if the listing needs to link back to a node. It is not much but I did find  away of making the link work. I like the fairly simple listing of the preview, a title and associated tags. The preview and title are obvious (and the listing could be extended by adding in a description as well.) but the tags could be used for categorising the collection folksonomically.

As well as Media, Media_Youtube, Media_Vimeo, I used Taxonomy and Views 3.

The first step was to create a video content type to take an embed url from the services and put it into Drupal and allow the user to add in a title, a description and a limited number of tags to define contents by creating a form and having the form save the node. The file URL needs to be saved as a file entity since that is what Media 2 now uses. When the form is saved, it creates a node with the url. Using the theme() functions in the templates (thanks to Finn at Ecobee.coop for helping me with this bit), the video stream can be shown using your own template. If you just use the standard node template which comes out of the Drupal box, then you do not need to worry about this at all.  So that sets up the individual nodes which have videos and descriptions of them in. But how to view them?

Well that is where views comes in. I created a new view call videos/all and used the fields to select all videos at that url. I also set up a contextual filter to query the videos which were set up with tags against them to filter on the tag term and return all associated video type nodes. The default is set to show all if there are no videos with a term.

The one issue that I did find with this was if I wanted to have the video itself linked to the node, not just the title (which can be set when you call the title as a field). I had to use the content:NID field which is excluded from the display.

In the content:Video field, I set up a link which referenced the content:NID  value in the link option, had it replace the spaces with dashes and use an absolute link to provide the URL so that a preview image could be used as a link as well like Vimeo or YouTube where you have both sorts of links in the galleries.

(Another long-winded way of doing this is using the global PHP and using drupal_lookup_path() to turn the node id into a URL but you shouldn’t need to do this. )

Using these simple modules, you can create a simple Video gallery that allows you to show videos from the various web services using the Media modules. I am sure that there are other ways of doing the same thing. I am sure that I will be exploring this more as I am hoping to use Media for a variety of other projects.

I have found myself spinning out in various directions due to existing work and personal commitments and things I am interested in. Some of these overlap in odd ways and some do not. I have largely stopped doing anyhting personal in the last few weeks for a variety of reasons but having had a couple of days off and looking at a backlog of unpublished posts on the blog, I’ve begun rethinking some of these. I have one or two things about Drupal and messaging to be published and to explore in the next few days. Admittedly these are slightly more work related but I am also interested in them.

I am also trying to do somethings for other people. I am admittedly way behind where I would like to be on these and have used the last couple of days off to have a think about how I tackle these and play ‘catch up’ to produce the work that  I would be happy showing rather than just trying to get a post out or a function finished. There are things that I can fold into work from these projects and I am determined to get on with them next. When I do, I can post about them.

As some acquaintances know, I have a notebook full of ideas which I have collected in the last few years. The intention of these has always been to build them into something useful and to explore how large scale things and ideas can be rebuilt for the individual. At this moment I have no idea how I am necessarily going to complete them but I do see the next few days and January trying to clear up things and tie off soome loose ends.

This does mean that I will have to rationalise the things that I will get involved with. Admittedly I overcommitted last year and I am trying to honour these commitments, but many of these are in areas that I am interested in. So decisions, decisions. I started doing some of the things that I do because I was deeply frustrated with a job that I was in and followed my own curiosity. It took me into new areas and langauges. I managed to build things but perhaps it came out of balance. I got frustrated because I could not build all the things that I wanted to.

I am hoping to go back to work refreshed and invigourated as well as recomitting to the current projects. I do not want to just finish them but build something with which I am happy. Some older projects may get rebooted in the scheme of things as they become relevant to work or other projects but they may also disappear. At this moment I amnot 100% sure what will happen to them but there is a rather dusty development section which needs spring cleaning. I may also move it to a secion called, tentatively, Kitchen Table, which comes from the notion of cottage working and also that fact that my desk is an old kitchen table.

I am excited but nervous. Spring cleaning, moving on is not easy and I am sure that I will see old projects and not want to get rid of them but I need to. I need to whittle away cruft and prune to renew growth and have fun again. I am also learning to sit on my hands and not to start anything new for a couple of weeks at least whilst I finish the current bits of work I have said that I would get on with.

I believe that most of these compliment each other and the problem that I am interested in should become clearer and more focussed in due course.

Onwards and forwards.

A quick Google search brought me to this post by Roy on building an HTTP service using Tornado. I am fairly new to Tornado but have been looking at it for an experiment at work. However I have been using Flask for other projects such as a quick and dirty RDF browser largely based on Chris Gutteridge’s PHP browser. (At some point very soon, I need to get back to this but other projects are slightly more pressing at the moment.)

After a quick upgrade to version 0.8, I have managed to put something together a little like Roy’s script but I’m hoping to go further and add a storage layer before tidying it all up.

Unlike Roy’s script, I’ve pushed the Tesseract and file handling code outside of the server. In the long term, I’d like to split out the file handling and storage facilities from the web server which means looking at the storage. As a quick step, I’ve popped in a link to a MySQL database but a far better option would probably be a NoSQL database like CouchDB or similar. I suppose a Key / Value store like Redis could be used as well (Redisfs apparently does something like this) as a back end. I’m keeping options open.

I do have a temptation to use RabbitMQ to notify various workers that a file exists which suggests that if I’m hoping to use this as a book scanner back end (discussed at the Textcamp event in August), then I need to add in an automated set of scripts which reads a directory and deals with the file and moving, storing and scanning them. Perhaps Tornado might be a long term answer but realistically it is not needed for a test project.

Also, Tesseract will need some training as I’ve discovered this evening playing with some newspaper text and seeing some of the results. As one of the reason I began this was to store old fanzines and newspaper articles which I’ve stored for research but are now degrading, that might be a problem.

Either way, this is a way of moving ahead with the book scanner conversation and building something small to scratch some itches.

One of the matters that did concern us was the word “data” and what this might mean to literature and humanities. One assumption that we had was that it perhaps had a more defined meaning to scientists. But what is data to humanities? Is it the manuscript, the notes, or the published work? We decided that ‘Work’ might be a better word for the overarching principle.

One of the issues that is important is re-use and subsequently closing the re-used work down and making it non-open. The major party that we had in mind was Google Books. Whilst they are making good and admirable strides in the digitising of out of print works but there is no API or metadata store that can be used to mix up the data or to mine it in any other way. Effectively we end up where we started: with a technically open text tied up in ways that cannot be re-used.

Re-use and re-mix are extremely important within digital humanities. Influence and building on works are central to movements like Modernism and also ensuring that works and authors are accessible. Works are adapted and take on their own lives or segue from such moments.

The final major point was that citations and the underlying cited text should be open. Whilst the core of the principles are about the work and ensuring that it can be worked on, a fair amount of work goes into notes and annotations to the text (such as the great Annotate It tool) and these provide a meta work for people to build on. It is vital for debate that these are not put into a closed arena, not just for the sharing of notes but also building on the notes. They might also be put together into a new work or an annotated version of a work put together to build upon the work with communal notes.

This does represent a step forward in open literature and digital humanities. I really hope that debate does start and that these can be developed and make concrete.

The ever useful Elegant Code blog has a quick guide to streaming files using Node that pretty much comes out to the box, or Github. Using some of the code and upgrading my own copy of Node.js to version 0.59 and reinstalling NPM, I started streaming a couple of MP3s off my laptop as a quick experiment towards something that might be of use to me. One of the ideas that I’ve been having is creating a streaming layer using Node to reduce Apache traffic.

This is the first stirrings towards a hack that I’ve really had a chance to play with recently but I’m hoping to get back into some real Node development again. It looks like there is at least one really useful FFMPEG library to do transcoding and produce thumbnails.

I needed to get some county data, and possibly other sorts, put into some forms so that it can be standardised. One of the issues that I’ve been trying to do is to make sure that data is clean across various systems.

One of the issues that has come up is the connection library to the Redis server. I’ve used Nicholas FF’s phpredis library and the Rediska library. The matters that I’ve been considering are the size and complexity of each library.

I do like phpredis but as it is a C library needs compiling so that puts a potential barrier to its use in Drupal. Also the user may not have root access to compile it. As earlier posts might show, I loved Rediska but it seems overly large and complex for such a slight task. I was running some tests today on another project and it did not appear to be closing its connections properly (an issue which might be ‘QDH’able in the short term). Both libraries are complete and large which is what makes them great Swiss Army knives for Redis and PHP.

But I would like something small and light that has access to a subset of commands and is pure PHP. I suppose a long term desire might be to implement some of the pub/sub commands but I cannot think of a use at the moment.

I’ve taken ideas from Salvatore Sanfillipo’s post which uses Ruby and ported these into PHP so the code uses Redis’s sorted sets to get the relevant items. I need to complete the code to see how it fairs under AJAX in Drupal. It would be a great sidetrack to follow some of the other possibilities such as search prediction, or perhaps this might go sideways into using that to pull up homonyms or synonyms where these are stored.

But that is definitely another day and another project.

One of them is the idea of autocomplete. Nothing new there but something that can be easily overlooked when looking at the data instead. User Experience (UX) isn’t the only reason I’ve been looking at this. I’m also trying to find ways of keeping some of our data cleaner than it is and limiting options to a validated set currently seems to be the best way of doing this. To that end, as I’m still in the early stages of learning Drupal, i searched the Drupal forums and came across this comment (http://drupal.org/node/1117562#comment-4452968) which has really helped me understand this. I’ve only implemented a cursory version of this code at the moment, throwing some rough values into the array to return, but the results appear to show fruit.

My intention is to back this onto a Redis store which will hold the cached data so that the site is not putting undue strain onto the main database and that the data store can be shared with other applications on the site. I’ve got the Rediska library which I’m currently using but it has far more functions in it than I’ll ever need so I might roll my own pure PHP stripped down library. I’ll see what time I have.

After a conversation at work, I’ve started work on a  content aggregation system to store feeds from Twitter for some of the associated feeds from work. The idea had been to allow some of the users to search older tweets and to push the users back to Twitter and the relevant timelines if they want to follow them.

As the feeds had moved in a recent update to Twitter, I did have a little trouble finding the JSON feeds. georgep83′s post on Software Hacks got me there. I’ve used some Node code knocking around in my /home directory that quite probably comes from a tutorial (I think one from reading Tweets and presenting them to the browser) but gives me a start in trying to create something to understand node.

I’ve also used the Redis code downloaded from NPM,  and popped the data into a store. I’ve adapted the code to parse a stream from my timeline and then parsing the JSON and stringifying it to store it into the key. What would be good is to be able to iterate over a list of relevant public Janet user names and to use the name as the key value and store data against each key. A further refinement (though might be one too far) is to add a time to each key so that it contains the tweets for the day. Looking at that sentence, I think that might be too far.

Anyhow, once I’ve got the iteration sorted I can build the parser to return relevant pieces of data to the Janet users. I might try and do this in Javascript as a learning exercise but I think I might be better off with something like PHP or Python. There is a large amount of data in the returned JSON now which presents some serious possibilities.

However I’m reading the Twitter API’s terms and conditions and offline use guidelines which are fairly fierce. I’d read that they were (and think I got caught in an earlier life on Twitter for JISCMail and got my desktop IP banned whilst writing a Perl script that got a bit over enthusiastic in a loop – oops!) but there is a lot to think about in terms of our own intended use. As well as offering some advice, it does offer some examples of how to use the data in our own services.

There are issues to think about within the web and marketing teams to ensure compliance with the terms. Much to think about and potentially getting into some serious waters… It does make me think twice about trying to use the Twitter API at all though.

After Book Hackday, I’ve written a post about beginning to develop the Open Correspondence toolkit for the Open Knowledge Foundation’s Notebook blog. I was also contacted regarding converting the TEI XML pages into JSON, which I am currently working on.

Once I’ve done some more work on it, I’ll post the code and more about it.

I’ve been working on another project which may or may not be open. It is certainly interesting but I am not sure I can say much more than that. I hope to have a blog post up soon about it but I am rather excited by it and its possibilities.

Meanwhile, the work project continues apace with some surprising outcomes for me. Following watching a video on Facebook’s architecture, I’m beginning to see certain parts very differently. I really do hope more on this but I’ve got some building to do and a bit more delving and reading that needs completion.