After a conversation at work, I’ve started work on a  content aggregation system to store feeds from Twitter for some of the associated feeds from work. The idea had been to allow some of the users to search older tweets and to push the users back to Twitter and the relevant timelines if they want to follow them.

As the feeds had moved in a recent update to Twitter, I did have a little trouble finding the JSON feeds. georgep83′s post on Software Hacks got me there. I’ve used some Node code knocking around in my /home directory that quite probably comes from a tutorial (I think one from reading Tweets and presenting them to the browser) but gives me a start in trying to create something to understand node.

I’ve also used the Redis code downloaded from NPM,  and popped the data into a store. I’ve adapted the code to parse a stream from my timeline and then parsing the JSON and stringifying it to store it into the key. What would be good is to be able to iterate over a list of relevant public Janet user names and to use the name as the key value and store data against each key. A further refinement (though might be one too far) is to add a time to each key so that it contains the tweets for the day. Looking at that sentence, I think that might be too far.

Anyhow, once I’ve got the iteration sorted I can build the parser to return relevant pieces of data to the Janet users. I might try and do this in Javascript as a learning exercise but I think I might be better off with something like PHP or Python. There is a large amount of data in the returned JSON now which presents some serious possibilities.

However I’m reading the Twitter API’s terms and conditions and offline use guidelines which are fairly fierce. I’d read that they were (and think I got caught in an earlier life on Twitter for JISCMail and got my desktop IP banned whilst writing a Perl script that got a bit over enthusiastic in a loop – oops!) but there is a lot to think about in terms of our own intended use. As well as offering some advice, it does offer some examples of how to use the data in our own services.

There are issues to think about within the web and marketing teams to ensure compliance with the terms. Much to think about and potentially getting into some serious waters… It does make me think twice about trying to use the Twitter API at all though.

As with storing the Open Correspondence data in CouchDb to prevent reparsing, I’m wondering if Node can be used as a server to get and store changing data from an interface in JSON and then storing it in CouchDb and letting a client know it has done all that. It would allow for a framework to be continually updated from different data sources.  Just an idea.

I’ve been working on Twittering RSS feeds for the JISCMail service and getting the service news feeds to become tweets using Perl using XML::FeedPP and LWP::UserAgent. I’ve even got a script reading Twitter and posting back any posts from the account to an email address so that the helpline doesn’t need to constantly log into update itself.

Clearly RSS on its own is not going to help with the constant stream of news attention required by some users. It does for most people I suspect who are not running in real time but messaging systems on the web are changing and it is getting faster which perhaps demands a rethink  of how silos, like Twitter and Facebook, and protocols, like RSS, work together.

I noticed that the pubsubhubub solution that Erick points to builds on Atom and pushes via an IM style solution. Andy Skelton at WordPress has developed a Jabber plug in (which I suppose goes some way to alleviating the problem but only for WordPress).

Pushing content and transforming it into a different protocol is the easiest way currently to make sure that news or events are ported into different services and that the community can be developed. Building and updating communities has never been easier or frustrating at the same time trying to see how the different services talk to each other and how to build “real-time” update when necessary.