Wordcloud of Defoe's journey taken at Textcamp

Using the Many Eyes Word Cloud, we can see that Scotland is unsurprisingly the largest item but also Lord and Earl are also popular, suggesting that he stopped with or met the aristocracy rather than just travelling randomly. Dave Flanders and John created some cool visualisations using the tool which allow you to follow words in the text and to see which are the most linked to words (using bigrams I would suppose) in a tree fashion. It is certainly something at I will be looking up later for “quick win” visualisations.

One of the intriguing projects that was suggested was building our own DIY bookscanner using links currently stored on the Textcamp 2011 wiki pages. I think that Dave Flanders might be organising a hack weekend to actually build the machine for real use. I find it interesting but thinking that it would be cool to also see if can be built at home or using iPhone / Android OSes which also entails a software hack, unless an app already exists. That is something to explore later.

Mark MacGillivray, of OKFN and Cottage Labs,  and Brian Hole of Ubiquity Press, spoke about Open Access and making scholarship open but also retaining its rigour. Using Open Access, we should be able to share the data, the ways of interpreting it and and the final interpretation which is published.

The science community has been doing this for some while and things like the Panton Principles and Science Commons are showing the way. One of the ideas was to write a handbook for how to use openness in literature and that it is something that we need address and build on. We ought to write an open guide / manual and build on / develop the Panton Principles where necessary as a core set of principles to work with.

Having days like Textcamp and Book Hackday are extremely useful to think about this and to work on the ideas. It is easy to get into echo chambers of mailing lists and blogs, we need these events to meet new people, be challenged to explain ourselves and to either build on the day or go away with ideas to test and try out. The day has excited me out using word clouds again and doing a bit more work on them as a tool to make them useful. It has also got me excited about book scanning and doing some hardware hacking (which I’ve not really done) before.

Running the Pope essay through Wordle makes me excited about testing what we can do with the ECCO TEI documents that John Levine  links to. Can we hyperlnk to other texts, author and events that are mentioned in it (not just with the annotator tool but in generated HTML) or use HTML 5 to embed audio links to further discussions or pronunciation (for example Byron’s Don Juan which has been argued as pronounced “Jew-an” rather that “Hwan” and the arguments for and against).

Perhaps that gets to one of the issues that arose in the break-out discussions in the kitchen. After the lightning talk about digital publishing, there seemed to be an argument about whether current digital publishing was really pushing the boundaries or flailing around. I do think that it has some real benefits for niche publishing but these have not been fully explored. The model will need to change and perhaps become more open in those senses, perhaps linking the raw data to the interpretation earlier to allow the relevant community to peer review the data earlier. Just a suggestion. There are two distinct communities, the top-down business layer and the grass roots layer, activists, data developers and so on. Both would appear to have broadly similar aims but how to put them together  in a useful way for both to learn. Don’t get me wrong here as I believe I’m at the grass roots layer, but I think that both sides do have a dialogue which could get around the issues that the music and film industries have found themselves in, i.e. confrontation. We are here to disrupt and make.because we are passionate.  We care about the industry. Publishing is an industry which needs to change and transform itself. Put the two together and there are ways of moving forward. My hope is that in future events, we could get some more publishers along to the event.

The other important thing is that these conversations carry on afterwards. The round table discussions where great as were the break-out in the kitchen ones but they need to carry on or we create our own echo chamber which reduces the value of what happened yesterday.

Whilst I did not do as much coding as I wanted to yesterday, I met some new people and caught up with colleagues. The fact that organisations such as JISC are supporting events like this shows their underlying importance and use to the community. We’ve started, now we need to carry on by chatting, blogging, sharing and doing more of these events.

I’m trying to move them onto the web and make the useful, not only in terms of reading about them but also to make them linkable. I’m trying to get them out of being placed on a web site as Word or PDF downloads and move them into being web pages with comments. Drupal 7′s inbuilt book module is probably the way to go and is producing some really nice results in the hacking I managed on Friday. There is a certain pleasure now in that I began the hack at 8:30 and within an hour, I had a working document (albeit I wanted to mess around with the URLs to make the nicer and far more meaningful). It had comments and was generally felt to be good.

The next task was to work on a way of doing Frequently Asked Questions (FAQs). Having begun some of the work using the Frequently Asked Questions module, I decided it had to many issues for us (including not being able to control where the page was and it did not appear to play nicely wiht the Pathauto rewriting module), I write my own content type which we can manipulate via the Views module to create sets of FAQs. When I’ve got more time, I may come back to the module and try to help fix some bugs.

Whilst neither of these are finished items, it was a pleasant day hacking and creating, getting prototypes ready in a day. I’m taking this as a sign of increasing familiarity with Drupal. I do, however, need to find a morning to finish the Sugar SOAP integration module and tidy that up. Ideally I’d trying to find a way of integrating it with the current module to offer swapable backends.

I’ve also started looking at using Redis for caching again in a major way to ensure that various static fields of data, such as UK counties, can have a common reference to reduce data cleaning issues such as county begin written as co., co and county. I’m also looking at the issue of Linked Data and how to integrate the ideas into our current projects. For now I’m rereading Tim Berners-Lee’s guide, linked from the linkeddata.org website and formulating ideas and refining the ones I currently have.

Ambition might bet the better of me but at least I feel like I want to take all of this on and to try to improve skills and learn more. In the meanwhile, I have some serious hills to climb.

Update:  This post has got me rethinking the Open Correspondence RDF and Linked Data. The more I delve, the greater my sense of needing to rethink that part of the project and to complete the correspondence links. Most of them are there but need complete linking. I also need to look at the Python’s RDFlib and perhaps make better use of the Sparql qeuries and stores. I sense an evening or several of experimentation before a hacking weekend to resolve these issues.

Firstly James Wilson, project manager of Sudamih at Oxford University Computing Services, outlined the Database as a Service (DaaS) project which I think outlines a desperately needed service. The project seeks to allow researchers to upload their datasets (I believe from SQL and CSV) into a MySQL, PostreSQL infrastructure with  a commion front end though with  access control levels to the data itself. The idea is to keep data sets available for long term use.

The second important point was that data sets need to be kept available if researchers move on, kept open for sharing since the same data can be used across the field or even by different disciplines or funding ends. Resources, as Claire Warwick of UCL, need to be kept available for the long term, partially in response to promises to funding bodies but also for citation purposes and re-use by future scholars. There are sites which are now appear moribund but could be kept useful if the data could be moved somewhere or the project kept in the service such as the above DaaS concept. Of course sites do need funding to stay alive and the notions of sustainable business models (from free to pay for access) were skirted over.

(I do wonder if it is practical to offer / build something like this as an Open Knowledge Foundation project as an adjunct for CKAN for smaller projects. But perhaps that is another post for another day…)

Jacob Dahl, of the Cuneiform Digital Library Initiative, was one of the only speakers who touched on the openness issue. He commented that there is a site which draws from his open database and makes some amendments but these are then not offered openly back to the originating site ir users un an open fashion. Again this leads to a “silo” mentality which prevents knowledge being shared and developed. This is a more insidious threat to developing datasets and databases since the knowledge cannot be easily shared. The scary thing about this is that rather than the websites being made moribund, the data itself is and a community cannot develop around it to refresh and maintain the data. Perhaps this is a more long term threat to Digital Humanities than tired-looking websites.

On a tangent, one of the speakers mentioned Alastair Dunning’s blog post on digitisation. I’m not going to summariuse as it is fairly short but the outcome that I take away is that digitisation is necessary but it needs to allow users to create new queries. This cannot happen unless the dataset is maintained and that access is given through APIs or search. (Funny how search comes back again. I’m sure it is haunting me.)

The afternoon was rounded off with a talk about the CLAROS project which  is using Semantic Web technologies to query several major databases of Classical Art across the world. It is something that I’m interested in (with the endpoints on OpenCorrespondence but I’m not quite there yet) and it marks the future for projects but I do wonder if the basics, the technological infrastructure for researchers needs to exist first. It comes back to the chicken and egg though. If the possibilities are not given and developed in prototype or early working models, then they remain only possibilities and not useful.

I believe that there are  a number of outcomes that arose and debate which  I’ve outlined above rather than talking about the individual talks. We come back to the notions of openness and preservation. I think that there a quite a few things that could be developed to aid researchers and also issues to keep in mind for developing future resources.

The search  is rather basic, a simple form to enter a phrase or words, and the results show the text and the letter url. On the list of things to do is to create an advanced form to allow the user to filter the results down further by date or to find relevance in the text.

From what I can see there are things that I can do on top of the simple search to achieve this. It would be useful to be able to cut the selection down by date which could be parsed from the text and anything not in it is discarded. Perhaps making the searches less naive and trying to discover relevance in the results. Perhaps there is somebody called Nickleby in the letters who is not part of the novel, ‘Nicholas Nickleby’.

Simply put there is a fair amount of data munging that needs to go on next. That’s fine.

The next step that I’m working on is the use of OFS to run some of the endpoints and XML streams that are used for internal purposes, such as locations or the RDF endpoint. I’m hoping to use it to bring through the Linked Data into the letters themselves. I’m looking at using these mainly for performance reasons. Along with a hack on the places that I’m hoping to do next week, the man body of Open Correspondence will be done.

Next up is better data munging and information extraction, such as rewriting the parser and adding more letters into the database. Essentially I’d like to provide better data in accessible formats for the letters and to perhaps offer some tools to kickstart development.

I’m going to the Research Databases in the Humanities workshop to see what else we can do with the data and the site.

A web of infinite information: does that sound like a scary problem of “just too much”?

in a “Mamas, Don’t Let Your Babies Grow Up to Be Data Wranglers” where he discusses an interview with Evan Williams on GigaOm. (I’m not going to discuss the interview here (but it is an interesting read).)

(I’m not sure that I can agree on the idea of the decentralised web being dead(I’m not sure it is) but the links between sites and services are becoming increasingly visible, sometimes deliberately so and sometimes because they are being used as a service. However I digress…)

In response to the Om Malik’s question: “You feel there is just too much stuff on the web these days?”, Williams responds:

There’s too much stuff. It seems to me that almost all tools we rely on to manage information weren’t designed for a world of infinite info. They were designed as if you could consume whatever was out there that you were interested in.

before identifying Twitter as a response to this and finding the signal in the noise.This response still requires the user to identify the signal that they are interested in and to follow it. Machine algorithms may not necessarily identify what a user is precisely interested in  but, from what I understand, they are getting better.

However we come back to the question that Tim Davies posed in the aftermath of the Open Data hackday. He identifies that that there are two approaches to Open Data data-led or problem-led. The data-led approach finds “some data of interest, and then explored what could be done with it.” and the problem-led is to start “with an issue to explore and then seeking data to work with”. Of course both have their issues (data-led can lose focus or problem led can stuggle to identify the relevant data).

This encapsulates the issue that Evan Williams identifies with the infinite amount of data on the web and how to make it useful. The approach still seems to be very much a data-led one rather than a problem-led one (though this is not to say that these approahces do not exist for apps or sites). The tools to usefull mine the existing data are only just being developed and it recalls something that I heard a while ago. The tools that we have now solve existing or previous problems but not tomorrow’s ones.  They are being developed by interested parties but somehow we need to get the data developers talking to the problem solvers to really make the existing data sets.

In the blurb to Phillip Janet’s Data Analysis with Open Source Tools, there is an appropriate line:

purpose is more important than process

So Marshall’s question might well be turned slightly around. Rather than looking at the notion of there being too much data, its one of how to identify the purpose to mine the noise for the signal. There is only too much information if you do not have a purpose in looking at it.

Al Power came up with the cash visualisation here : http://www.d1080072.cp.blacknight.com/hackday/ and Andy Cotgrove came up with the cool Tableau visualisation (http://public.tableausoftware.com/views/Artsfunding/Fundingamountinteractive?:embed=yes&:tabs=yes&:toolbar=yes) using all five years of available data on the site.

I played with tag clouds to start diving into the data itself which links to further and more detailed data (which needs some serious work) and also Simile to create a timeline but I’m not 100% sure that this is successful really.

Since the hack, I’ve added the complete data set that Andy Colgreave put together to the database. I’m now trying to build a front end to it and also to create the dataset in XML and JSON to add it to CKAN.net. Perhaps the dataset is probably the easiest task and the front end can be developed in time.

I’d like to add the Heritage Lottery Fund data where is applies to arts or data from the opencharities.org website to try to build a better picture of the funding a projects that are happening.

It might take some time…

Update: Tim Davies has got an thought-provoking post on the day

Opening a “one box wonder” Xapian search in Python is done via:

xapian.WritableDatabase(db_path, xapian.DB_CREATE_OR_OPEN)

where db_path is the database name that you want to give the index and open the index using:

xapian.Database(db_path)

Since the project uses Pylons, the controller used a path out to the .ini file loaded at runtime to link to the correct database.

Using the documentation on the Xapian site for remote backends and the Python bindings, I was able to quickly adjust the code so that xapian.WritableDatabase is replaced by:

xapian.remote_open_writable(“<host name>”, “<port number>”)

and is opened by:

xapian.remote_open(“<host name>”, “<port number>”)

Once that is set up, then all you need to do is to start the the TCP server which is what I’ve been looking at. I downloaded the tar.gz file of Xapian-core from the Xapian site, configured and made on Ubuntu Lucid Lynx and then ran xapian-tcpsrv –port <port number> <database name> in a new terminal window which allowed me to test the connections and get them ready for going live.

Changes are afoot on the Open Correspondence site as well. As part of a conversation that involved Keith Alexander, of Talis, the project is going to evolve into a slightly more Linked Data direction with references to the books, magazines, correspondents and so on. I’d already started going in this direction with the correspondent links (such as http://www.opencorrespondence.org/letters/correspondent/Miss%20Hogarth) so this is really an extension of where we need to go to connect to other resources such  as Dbpedia, Wikipedia and so on. The fact that it is Dickens’s bi-centenary in 2012 gives an added boost to the project. The Linked Data approach gives us the chance of creating some sort of framework for future expansion and linking together of data sources, not only at a literary level but also socially. It also encourages me to sort out the content negotiation work that was started and to try and follow the FAQs that the Pedantic Web group have posted to make sure that the site follows the best standards that it can and to build them into future developments and directions.

As with storing the Open Correspondence data in CouchDb to prevent reparsing, I’m wondering if Node can be used as a server to get and store changing data from an interface in JSON and then storing it in CouchDb and letting a client know it has done all that. It would allow for a framework to be continually updated from different data sources.  Just an idea.

I’ve nearly finsihed the ongoing accounts project at work. The framework is up and it went through testing over the last couple of weeks. There are a few rough edges and some bugs which still need fixing but it largely seems to be there now.

I’ve also installed the first part of a messaging server written in PHP (taking ideas and concepts from JMS and Python’s Routes for service urls) which takes a message from the core CMS system and routes them to the correct service using SOA. If there’s an issue with the service then it logs it and queues the message using Redis (athough an array might be quicker, I wanted the queue decoupled from the server if it failed or had to be restarted and the memory was wiped). I need to finish up the worker to dequeue at certain points in time but it is expected that I’ll get it finished in about four days once I’m back at work.

I’ve done one or two things on the Open Correspondence site as well. I’ve tidied up the source XML and the sources XML as well to expose them so I need to update the site itself. The next thing I think we need to do is to start writing stuff to expose the underlying data and to show what you can do with the data. One of the things that I want to do is to write a function which I can put behind either Protovis or Javascript Infovis Toolkit to convert a SPARQL query into the relevant JSON and I’m thinking of using Lee Feigenbaum’s sparql.js script. Quite possibly I need to write some sort of API to the dataset to allow other queries to be run.

My friend, Simon Biles who owns Thinking Security, and I have been talking about a Knowledge Management project which is slightly aligned with some stuff I’ve been thinking about storing research pages for RSS and web pages. He’s thinking in terms of MS Office documents which means a little investigation into the various types of structured storage in Office and the ways that Office has changed to mine different types of documents. It does appear at first glance though that newer versions of Office and Open Office are similar in terms of finding the metatadata being collections of XML documents in an archive.

I’m just starting the research on a new book and looking around for any open source / free software to capture a url, mark it with the time accessed (for later bibliographical purposes), capture the raw HTML, and possibly allow me to tag it for folksonomical reference if I want. What would be sort of cool is to have an interface to share the results later or just post an XML / RDF file to be posted later.

I suppose what I essentially want to find is something along the lines of a moleskin for electronic notes? I can see various subscription services listed but I really want something on the desktop to create  a relevant project archive to later share. Potentially this does add to the issue of lots of mini-silos by creating more but if , in Bibliographica style, they could be linked or linkable, I think it could be an interesting way of sharing research links or allowing bodies to create a meta-frame calling from the shared resources.

I think that this falls into the realm of archiving, which poses issues in the UK, especially when it concerns commercial sites as my reading of the consultation has it. Wired UK has an article on the issues of archiving web sites in Britain and the legal difficulties therein. The British Library has been working on an archive (including some from shops no longer extant) but can only archive the site if the copyright holder has given permission. Even the consultation paper (itself archived now) is vague on this.

Ultimately this will hobble research if ways of noting and sharing the relevant data and metadata cannot be found to allow sharing and relevant notation. It would also mean that I’m left to the vagaries of my browser or remembering to make a note of the link in a new moleskin.

Building something along the lines of what I want might create a tool which other people might find useful.