Personal Computing: The Whole Of Twitter In Your Hand

A quick check on Twitter this morning shows me they're up to message number 1,179,118,180 or just over the 1.1 billion mark. That's a pretty big number - or so it seems, but in the context of data storage devices, it's not that big. Let me explain...




Assume Twitter messages are all the full 140 characters long. That means, assuming all messages are being retained, that the whole of Twitter is approximately, 153GB in size. OK, so there will be data structures needed to store that data, plus space for all the user details, however I doubt whether the whole of Twitter exceeds 400GB. That fits comfortably on my Seagate FreeAgent Go!

If every message ever sent on Twitter can be stored on a single portable hard drive, then what on earth are we storing on the millions of hard drives that get sold each year?

I suspect the answer is simply that we don't know. The focus in data storage is to provide the facility to store more and more data, rather than rationalise what we do have. For example, a quick sweep of my hard drives (which I'm trying to do regularly) showed half a dozen copies of the Winzip installer, the Adobe Acrobat installer plus various other software products that are regularly updated, for example the 2.2.1 update of the iPhone software at 246MB!

What we need is (a) common sense standards for how we store our data (I'm working on those), (b) better search and indexing functionality that can make decisons based on the content of files - like the automated deletion of defunct software installers.

There's also one other angle and that's when network speeds become so fast that storing a download is irrelevant. Then our data can all be cloud-based and data cleansing becomes a value add service and someone else's problem!

Redundant Array of Inexpensive Clouds - Pt III

In my two previous articles I discussed Cloud Storage and the concept of using middleware to store multiple copies of data across different service providers. In this final part, I'd like to discuss the whole issue of security.

Using "the cloud" to store data requires a major shift in thinking; traditionally all your information would be stored locally and therefore benefit from the advantage of physical security. Not only would someone need to hack your firewall to get network access, they would then have to obtain system access too, and likely as not would be spotted (hopefully) quite quickly. So, retaining physical access to data has been a significant benefit.

Now we've obviously been trusting a form of cloud storage for some time. Email systems like Gmail, Hotmail and Yahoo have always had access to our email data and have provided limited storage capabilities but they haven't really been the foundation for running a business (although I'm sure there are organisations that have done it). Putting data into the cloud means there's always a risk of someone else getting to your data. You make someone else the guardian or gatekeeper of that data access and rely on the quality of their encryption and access controls. So, it is important to understand what facilities each infrastructure provider offers.

Amazon Web Services

Amazon have a great whitepaper on security, which can be found here. It highlights the level of physical security offered (which is high) plus details of the logical security of data. It may seem surprising that Amazon don't routinely back up data on AWS but rely instead on multiple copies in remote locations, however backup and archive should be thought of as distinct requirements. In addition, data at rest in AWS is not encrypted; users of AWS should therefore ensure their service provider offers this capability at source.

Nirvanix

Nirvanix have two white papers which discuss data security. They can be found here (registration required). As with Amazon, Nirvanix are keen to highlight the security of their facilities and adherence to Statement on Auditing Standard (SAS 70) certification. They also go further in indicating that data is stored using RAID-6 and RAID-10 protection, with backups in place too.

Summary

Both AWS and Nirvanix offer good physical security and SSL encryption for data in flight. Encryption at rest and backups are not routinely offered and therefore a cloud user should weigh up how these features are to be implemented. This takes us back to the original premise of these postings, the idea of using multiple cloud providers to add resilience and availability to cloud stored data. It also demands a set of standards for cloud storage use, which I am working on even as I write this post. Watch this space.