I’ve been meaning to get into doing some work using Node.js for bits and pieces. Node offers some serious possibilities for server side programming without using cron jobs, creating event driven programmes that reduce load on the CPU (not using cron jobs) and for more agile and responsive backends to be created.
After a conversation at work, I’ve started work on a content aggregation system to store feeds from Twitter for some of the associated feeds from work. The idea had been to allow some of the users to search older tweets and to push the users back to Twitter and the relevant timelines if they want to follow them.
As the feeds had moved in a recent update to Twitter, I did have a little trouble finding the JSON feeds. georgep83’s post on Software Hacks got me there. I’ve used some Node code knocking around in my /home directory that quite probably comes from a tutorial (I think one from reading Tweets and presenting them to the browser) but gives me a start in trying to create something to understand node.
I’ve also used the Redis code downloaded from NPM, and popped the data into a store. I’ve adapted the code to parse a stream from my timeline and then parsing the JSON and stringifying it to store it into the key. What would be good is to be able to iterate over a list of relevant public Janet user names and to use the name as the key value and store data against each key. A further refinement (though might be one too far) is to add a time to each key so that it contains the tweets for the day. Looking at that sentence, I think that might be too far.
Anyhow, once I’ve got the iteration sorted I can build the parser to return relevant pieces of data to the Janet users. I might try and do this in Javascript as a learning exercise but I think I might be better off with something like PHP or Python. There is a large amount of data in the returned JSON now which presents some serious possibilities.
However I’m reading the Twitter API’s terms and conditions and offline use guidelines which are fairly fierce. I’d read that they were (and think I got caught in an earlier life on Twitter for JISCMail and got my desktop IP banned whilst writing a Perl script that got a bit over enthusiastic in a loop – oops!) but there is a lot to think about in terms of our own intended use. As well as offering some advice, it does offer some examples of how to use the data in our own services.
There are issues to think about within the web and marketing teams to ensure compliance with the terms. Much to think about and potentially getting into some serious waters… It does make me think twice about trying to use the Twitter API at all though.
No Comments