Last week, whilst searching around, for the umpteenth time, for an application or website that would help me search through my tweets, I came across a wonderful web application called Tweet Nest. 

Made by Andy Graulund (who has a lovely visual blog), it is a beautifully designed web application designed to work on your own LAMP stack. Following a very quick and easy-to-follow install. Tweet Nest grabs your tweets and your profile and stores them locally in a MySQL database and allows you to browse through them using a simple interface. 

Of course, you don't need to expose your archive to the web, as I have done, and can instead choose to run Tweet Nest locally on your own computer. This comes with its advantages and disadvantages, the most glaring being that you need to have your computer with you whenever you want to use your archive of tweets.

Sadly, because Twitter only exposes your last 3,200 tweets via the website and the API, you'll only be able to retrieve your most recent 3,200 tweets. At the time of writing, I've amassed over 4,500 since January 2007 when I signed up, which means that over 1,300 of my earlier ephemeral musings are, for the time being, unavailable to the general public. Twitter say they'll eventually give us access to these tweets.

For now you're safe, and despite the fact that this is probably no bad thing right now, I can think of at least two main reasons why you'd want to keep hold of your entire twitter history:

Twitter won't go offline tomorrow... right?

I'm sure many people thought exactly the same thing about ma.gnolia. The number of factors affecting an application's availability is probably as huge as the amount of faith each of us puts in the online services we use on a daily basis. 

While Twitter is unlikely to go out of business tomorrow, there's always a possibility that they'll be acquired by a bigger fish at some point in the future - a fish whose privacy policies and track records make it difficult for some of us to continue using the service.

Tweet Nest means that, in the unlikely event that my tweets disappear from their original source overnight, I'll at least have all of my tweets up to the last half hour. 

Cloudy, with a chance of missing tweet history

When Twitter Inc. acquired Summize, I believed that Twitter's search engine would become much more useful. Sadly, there's still no built-in way to search your tweets further back than 10 days in the past, so a huge draw for me would be an application that allows me to do just that - search through my older tweets.

On a number of occasions, I've used Remy Sharp's excellent Snap Bird service to attempt to look into my twittering past. Snap Bird is, essentially, a version of Tweet Nest that's built to run within your browser, powered by JavaScript. It's also open source, just like Tweet Nest, which means that if you wanted to tweak it, to use HTML5 local storage for instance, you'd be able to.

Sadly, because there's limited error handling built into the app - something you'd be able to fix, if you wanted to contribute to the project - it becomes frustrating when the Twitter JSON API craps out midway through a search. This happens to me at least once every time I use it, something which probably won't with Tweet Nest.

Fire and Forget

Finally, once set up, with the addition of adding a couple of cron jobs, I never need to worry about the application again - bar updates, of course. By adding two lines to my crontab, my tweets are fetched every half hour between 6am and 1am, and my user profile is updated once a day.

If you're not familiar with the syntax necessary to create a cron job, just follow these instructions once you've installed Tweet Nest:

• Open up a terminal window and, if your solution is hosted on a server, ssh into your server using your normal account.
• Type crontab -e and hit enter.

• Add the following lines to the text editor that apepars

  0,30 0-1,6-23 * * * php /tweetnestpath/maintenance/loadtweets.php > /dev/null 2>&1
0 0 * * * php /tweetnestpath/maintenance/loaduser.php > /dev/null 2>&1

  Note: Obviously, you'll want to replace /tweetnestpath/ with the actual path to your installation.)

• Save your changes (usually CTRL-O) and exit the editor (usually CTRL-X)

Your new crontab will now be installed, and you should find your new tweets added to your local database every half hour. This is what works for me, but you may find that your server won't allow you to run custom cron jobs. If in doubt, you could always ask a bunch of people who know what they're talking about.

Looking forward to looking backward

I've only been playing with Tweet Nest for a few days, so I'm still finding things out. Already, however, I can see that it's a well made piece of software and I'd recommend it to anyone who values their tweeting history.

If you're not quite ready to take the plunge yourself and would just like to take a look at how it works, head over to http://tweetnest.abitgone.co.uk.