Computer Storage: Larger, Cheaper, and More Problematic

• By David W. Dodd
• 03/01/09

Information technology has evolved rapidly through advances in areas that include networked communication, graphical user interfaces, and processor speed. However, among the most significant advances contributing to the current power of information technology is high-density, random-access storage. Although the most familiar type of storage is the hard disk drive (HDD), various varieties now in use include flash memory, CD/DVD, and the emerging solid-state disk (SSD). In addition, numerous companies are now offering online hosted storage, often advertised to be “free” and “unlimited.” Along with the advances in storage technologies and the concomitant greater reliance on digitally stored information have come numerous challenges for institutions related to security, records retention and regulatory compliance, e-discovery, administration and reliability, and other factors.

It has been estimated that HDDs have improved by approximately 1,000,000 percent in the past 20 years, or about 67 percent cumulative improvement per year during that period. In the mid 1980s I bought a Zenith 151 desktop computer with two 5.25-in. floppy drives. I was ecstatic to eventually replace one floppy with a five-megabyte (MB) HDD.  Recently, Seagate and Western Digital, among the leaders in HDD technology, both introduced two-terabyte (TB) internal hard drives for desktop computers. The price for these is in the $200 to $300 range, making them readily affordable for home computers and media servers. For reference, 2TB of data storage is more than 400,000 times larger than my 5MB drive, and would hold roughly 200 digitized copies of the entire unabridged print edition of the Encyclopedia Britannica, or nearly 500,000 songs in MP3 format totaling 32,000 hours of music.

Space Is Up, Price Is Down
As capacity and performance have increased and prices decreased, the insatiable demand for storage has grown symbiotically. IDC recently reported that the demand for disk space has increased at a sustained rate of 60 percent per year. Users have come to expect unlimited data storage in a way that is unsustainable by most institutions. Companies offering ostensibly “free” and “unlimited” hosted storage have prompted misplaced comparisons to the professionally administered, secured, and reliability-ensured data storage provided by universities. This has led to uninformed suggestions that “several of those cheap 2TB drives could easily take care of our data needs… why not just install a few?”

Whose Data Is It, Anyway?
The problem with this notion is, of course, that it fails to take into account several critical points. First, a number of the “free” storage companies have come and gone. In some cases, users’ data went the same way as the defunct companies after their business models proved nonviable. As seen in news headlines, that little question of who actually owns and has access to your data stored on hosted systems is also a concern, particularly recently.

Data security is a critical concern, particularly for storage devices on users’ systems such as high-capacity HDDs, flash drives, and DVDs. Again, frequent news reports give an indication of the scale of the problem involving compromised data on these media. The Natural Resources Defense Council reports that in the United States alone, about 130,000 computers are discarded every day, and more than 100,000,000 cell phones are discarded annually. The amount of potentially compromised data these discards may represent is staggering. Newer technologies — including solid state drives (SSDs) and encryption tools — may help alleviate some security issues, but they won’t address the responsibilities that institutions ultimately have concerning regulatory compliance. 

A very serious problem for institutions today is the legal responsibility for storing, indexing, retrieving, securing, and ultimately disposing of data effectively. Although the ultimate disposition of e-discovery responsibilities is still working its way through the courts, it is fair to say that institutions have the responsibility for knowing what information they possess, producing that information in a timely manner when compelled to do so, and ensuring the integrity of information in their possession. This responsibility is well beyond the current capability of many institutions. Clearly it is nearly impossible for information stored in numerous, unsecured forms such as users’ media and devices, and in online hosted storage. It is also impossible on cheap HDDs that are not subject to backup and security protocols.

Planning for the Long Term
As storage technology evolves rapidly and is paralleled by what is proving to be an insatiable demand for stored information, institutions must tread very carefully. Short-term decisions will inevitably have a compounding effect. Planning and policies are essential in dealing with the long-term implications of digital information storage, and corresponding institutional responsibilities. Effective, well-informed strategies should be established and maintained. In many cases, this may require positions dedicated to this critical function. On second thought, digital storage may not be cheaper today after all.