Alison's take on digital storage

Yay! Alison the Archivist is back, with a post on digital storage: 

First things first. I want to make it clear that digital files can be maintained over the long term. Digital information is not hopelessly ephemeral.

I believe that most of our current misgivings about the longevity of digital data spring from the fact we have historically treated them as short-lived, transient objects. We make copies of files willy-nilly all over our computers. We send them all around the world via email. Digital storage is inexpensive, so when we run out of room, we just buy more—a luxury that we do not have in the physical world. We do not often consider digital files even to be the same type of animal as paper letters or paper photographs.

For many of us here in the Moxieverse, though, we now recognize that we possess digital files that are neither disposable or ephemeral. We are creating unique and indispensable records of our children's youth with our digital cameras. We do not have space to store all of our children's glorious artworks, so we scan them, hoping to find a place for them in the digital realm where none existed in the physical world. What are we to do? How are we to ensure that some of this precious evidence of a time now passed will persist?

First, we should be clear about our timeframe. How long do we want these files to last? You'll note that archivists rarely (if ever) use the words "forever" or "permanent." We are too privy to the effects of time on information. Instead, archivists consider how things might best be kept for the long term, or on an on-going basis.

I make this distinction because it is important for all of us to set our expectations at a reasonable level. What we are trying to do with both our most precious physical and digital records is keep them in the best possible manner so that they will last as long as possible. This is not the same as expecting them to last forever—a quite unreasonable expectation. We will do the best we can, knowing that our descendents will take up the same mantle and make their own decisions about how they wish to proceed.

Enough with the theoretical ramblings.

We all have a pretty good, innate understanding of what paper can and cannot be expected to do. We've used it our entire lives. It has been passed down to us by the past and we have seen how it deteriorates. On the other hand, digital files often seem hidden, tricky—the domain of other people "who know more about such things." Given the vast number of digital files that surround us, now is the time to shed such preconceptions. The moment has come to acquire a better, basic understanding of what digital files need to thrive so that we can move forward without trepidation.

So, how to proceed? Well, unfortunately, the one thing we should not do is to apply this innate understanding of how best to store physical objects to our decisions about storing digital objects. Let me explain.

In order to increase the life expectancy of a piece of paper or a photograph, the most prudent thing to do is to place such an object in a very cold, very dark, humidity-controlled environment. In fact—if you will allow me to exaggerate my point here for effect—to make a photograph last for the longest period of time possible, it would be best if you were to keep it in its cold, dark place without ever looking at it or touching it. Looking at a photograph exposes it to light which degrades the print even just a little, while touching the photograph exposes it to all the dangers inherent in physical manipulation, such as oils from the skin being transferred to the print or any accidental ripping or bending.

This is the exact opposite of the way that we should treat digital objects if we want them to last for the longest period of time possible. We need to open them, look at them on all sides, make sure they are still readable. It is not wise just to dump all of your most precious digital files on CDs or DVDs and put them on a shelf in the same way you might store a box of old photographs. The photographs do well in the dark. Digital objects do best in the "light," and the reason to keep them visible is two-fold.

First, as I mentioned in my original post, the physical medium on which we store our digital files is as subject to degradation as everything else in the physical world...CDs, DVDs, magnetic tapes, hard drives, whatever. Every type of digital storage has a panoply of ways in which it could fail, just like all physical objects. The scariest part of this process is the fact that it is not always patently obvious when digital storage devices are on their way to failure. Like your car, sometimes they give hints of an upcoming breakdown, but other times, they just die in the middle of the road. But when digital devices do fail, they tend to fail utterly.

The second reason that we need to maintain an active connection with our digital data is that both the file formats and the storage formats we use to maintain our digital information periodically become obsolete.** As mentioned by one of the commenters to my original post, if you have all of your files on old iOmega Zip disks but you no longer have an iOmega Zip drive, no matter how perfectly those disks still function, they will you no good if you cannot retrieve your information from them. As for old file formats, did anyone here use PeachText back in the day? Trying to open those files now is a tough prospect. Commodore 64 files with no Commodore 64? If you don't have the hardware or the software to read your data, you don't actually have the information at all.

So, in summary, the longer you leave your digital files alone without looking at them, the greater the risk that the information will not be useable when you go back to find it.

Scared? Well, don't be. Again, just like everything else in life, maintaining your digital records is a question of risk management. Ask yourself how much effort are you willing to put into the process of maintaining your data for the long term. You make these types of decisions already with your physical stuff, you just may not think about it so much because of your innate understanding of the way the physical world works. For digital stuff, we probably have to be a little bit more proactive for the moment, but, hopefully, this will all become old-hat in the future.

OK, I hear you all saying, bring on the practical tips for this balanced approach towards digital recordkeeping!!

As with my last post, I have no suggestions that involve actual brands or products. I'm afraid you will all have to evaluate what's on the market for yourselves. Moreover, you are not required to follow all (or even any) of my suggestions. Pick which ones fit into your lifestyle. Adapt them. Riff on them. Feel free to decide that you think all of this is too much bother and accept the fact that you may lose digital information. Or, feel free to follow them all to the letter, and know that you still might lose some data, just like you know you may lose those precious photos in your attic due to unforeseen circumstances. Whatever you do...pick a system that you will be happy to maintain, and don't just follow someone else's mandates. This is your information we're talking about. You get to decide what to keep and how to keep it. I am hoping to provide you with information that gets you thinking.

1) Controlled redundancy is a very good thing. You should consider storing multiple copies of your digital records on separate storage devices (say, on a designated hard drive at your house, and also a set of CD's stored off-site). By doing this, you will greatly reduce your risk of loss, mainly because you have made two copies—no other fancy reason. Remember, though, keeping your files in two places means that you need to keep your eye on both copies on an ongoing basis.

2) I like to use hard drives for my long-term storage because I don't like to spend hours plopping CD after CD into a drive to check each and every one of them.

3) Whatever type of storage medium you've chosen to use—say a hard drive—you should seriously consider copying your data onto a new hard drive every once in a while, say every three to five years, in order to help ensure that the physical medium is as fresh as possible. The same goes for CDs, DVDs...anything. And, do feel free to use the phrase, "I'm currently refreshing my digital storage media" in conversation. It impresses people.

4) When you do refresh your digital storage media, or indeed at any time that pleases you, take a moment to look at what you have stored there. Does it include everything you want it to include? If not, add or remove what you desire. Do you still have all the hardware and software needed to read and understand all of your stored information? If not...lesson learned. If so, do you foresee having the necessary hardware and software available the *next time* you do this operation? If not, update your choice of storage medium and/or decide which files should now be saved in newer file formats. When you do this, make sure that all of the information you were once able to see and use in the old files can be seen and used in the new ones.

5) Do not delegate the responsibility of storing your files to any online company who does not promise to take care of them until you tell them to stop. This includes Flickr, Snapfish and all those kinds of sites. If and when these guys go out of business, *poof* there goes your data.

Over time, we may find that we don't have to go back and check in with our data so often. But I don't think we'll never be able just to put digital data into "cold storage." I also think it might be clearer now why we are finding that digital files are more expensive to store over the long term. When you wish to ensure that your digital data is usable on an ongoing basis, it is much more labor-intensive to maintain, both mentally and physically, than paper records.

Finally, allow me to mention that there are those who feel that, given the reality of this situation, it might just be best to keep every single one of your digital files. These folks suggest, "Why think about it? Just backup the whole lot. Storage is cheap and we can just search the text of the files to find what we need." This solution may work for periodic computer backups, but it is a risky way to move forward with the information we want to keep for the long term for a variety of reasons. Let me just offer you offer one or two. How do you find a picture with a text search? If you save every digital picture you ever took...how are you going to find the one of Aunt Sara? Finally, if you never take the time to remember what it is that you've stored, you may very well end up with a lot of useless ones-and-zeroes because you no longer have the software to interpret your files. You may certainly choose to follow this path if it pleases you; I just want to point out its risks.

How do you pick what to keep and what to purge? Again, I'm afraid that's up to you. These kinds of decisions are made in the precise same way for both the digital and physical worlds, because, at its heart, you are dealing with the same thing: collecting, expunging, remembering, forgetting, embracing, letting go, thinking about the future, coming to terms with the past. Information  is meaningful.

That's a lot of words, I know, and I haven't even answered all of your questions. I invite all of you to discuss this further in the comments, and I especially invite responses from my fellow archivists...I know they'll have more to say on this topic!

Alison

**Those of you who have old slides might have just perked up your ears. Yes, regular-old slides are indeed becoming obsolete in just this same manner. Kodak no longer makes slide projectors and it is becoming harder and harder to find a working machine. If you wish to enjoy looking at your slides, therefore, reformatting them as digital files makes an awful lot of sense. But please, consider keeping the old slides once you have had them scanned. In future, you may wish to have them scanned again when the technology improves. Think of your digital scans as copies of the original, not as a replacement

Comments

[LPV]

I'm not an archivist, but a computer scientist. I just wanted to comment on choosing a format to store data in. It might be worthwile to chose to save your data in an "open format": http://en.wikipedia.org/wiki/Open_format because these tend to last longer than formats that are connected to a particular tool. For example, if you have important text information to store, you might not want to trust that Microsoft keep their ".doc" document format for Word consistent in years to come. Therefore it would make sense to convert your files either to plain text, if the files contain only text, or pdf if the files contain images/important formatting as well. That way, you will be able to open you files with a great many more different programs and the format is likely to stay more consistent in the future. I'm not saying the the ".doc" document format is likely to change, this is just an example :)

[songbird]

Another note about formats from another computer scientist. There's safety in numbers. The widespread use of the .doc, .mp3, or .jpg means that appropriate conversions will exist when you need them to exist. It might not be fun (or free) to open every document and "save as" the Format of the Future, but it will be possible, even though .doc and .mp3 are not open formats. Also, open formats are more useful when big companies get behind them. The OpenOffice XML format has been, up to now, pretty useless. But now that Micro$oft is switching their standard to it (according to latest rumors), it will be much more widely supported. The ogg vorbis music format is superior to mp3, but because iPods don't play them, they are not useful to most people.

[Cecily T]

Here's my question about this, like LPV says, you have to watch the format, but along with that, there's the metadata that comes with stuff like images. Yes, I'm a computer geek. Lots of programs can categorize images (Adobe Album, Microsoft's Photo software, Picasa) or tag them like Flickr. How do you hold on to the metadata, which DOES make it possible to search for "Uncle Matt" in images if you've taken the time to tag them?

[Eva]

Thanks for this!

My husband (who cares deeply about this topic) had this to say in comment:

Mostly good, but has some points that people keep repeating which simply don't hold.

If you have a lot of files on a format becoming obsolete (e.g. the zip disk) you'd be a fool not to convert it over when it's becoming clear that the demise is near. Even so, there are services that'll read just about anything from the old days for a fee.

As for file formats changing and becoming inaccessible, it's virtually an urban legend. For some reason people keep bringing up the 1980s (or so) .PICT format that Macs used as an example of something that "can't be read". Guess what--Photoshop out of the box reads it just fine, as do a dozen other image editing programs.

We're using the Amazon S3 service for backups; they're not going anywhere anytime soon. Also, since it's for backups we still have our local files (on two harddrives) should the need for putting the files elsewhere arise.

[pennifer]

Is it just me, or is Ask Moxie's format a bit wonky right now? When I go to the home link, I get only the page's template, no content. Yesterday when I visited the site, the content was all here, but only by scrolling down down down down past the template. Firefox just did an update, so maybe it's that...

[Simone]

Nope, Moxie's site looks just fine to me, but then again I don't use Firefox.

Thanks, Alison. This is great information to consider. I'm currently in "save everything" mode, which at some point I may want to change. But, at least I am saving multiple places (everything on my computer's hard drive, plus I back up onto an external hard drive every night, all photos go on CD, plus I have them saved on Costco's site).

[inki]

This is great information, thanks for taking the time to put it together and explaining the issues and options so clearly!

@pennifer: I've had that happen with Firefox too on a couple of occasions (not just Moxie, but other sites too), and restarting Firefox has fixed it for me in the past.

[Alison the Archivist]

@Cecily T: Ah, yes, the metadata issue. What will become of all of our tagging? For those of you who may be less computer-inclined, "metadata" is the information that both humans and computers attach to digital files in order to describe and/or identify them.

If you do plan to spend a lot of time attaching labels/tags/metadata to your images, you might want to take the time to choose a program or service that allows you to export that information into a common format (the most common would probably be .txt). Then, should you need to switch software packages or services in the future, you'll have a good chance of being able to import that information into your new solution. You can also look into embedding your metadata right into the image itself, although this has another set of preservation issues.

@Eva: I’m glad to hear that your husband is happy with Amazon S3’s terms of service, and am also pleased to hear that you guys are also saving multiple local copies of your data. You’re very lucky to have such a technical family!

That said, I’m not sure I’d call people “fools” for not migrating away from technologies that are becoming obsolete. Sometimes, time just gets away from us and we haven’t had a chance to put a system in place to maintain our data. To be sure, this might end up coming back to haunt us, but many people have a lot on their plate and data storage just isn’t always that high on the list.

I think it will also be nice to see the day when the average family doesn’t have to rely on paying other people to get a hold of their own information. I am of the firm belief that, with a little foresight, every family can not only maintain a backup copy of their digital files, but also a more carefully guarded digital archives, even if they are not so fortunate to have someone around like Eva’s husband.

[TheMacMommy]

I've just started researching this but I was wondering if anyone has any suggestions on how to preserve or archive a blog? I mean the whole thing? Is there some kind of service that might convert it into a book for you or a way you could archive the text? I was thinking of just printing PDF files of the pages and making it into a book that way. Would that work? Anyone done this?

I wish my grandmothers or mother would have kept journals like we bloggers do today. Maybe my kids and grandkids will feel differently but I would love to be able to read journals if they wrote them. There are letters from war times here and there but the handwriting is so hard to read and the paper or course is aging. I wonder how our blogs written for our kids will be 'delivered' to them in the future.

I feel like Michael J. Fox all of the sudden! :)

[Archivist Alison]

@TheMacMommy: Printing out information is often a good thing, but let me ask...Do you link to many external sites? How much of the information on your blog do *you* actually control? Otherwise stated, does your blog still make sense when decontextualized from the Internet?

If not, figure out what your blog will look like and will accomplish in whatever physical form you choose and decide if you are OK with creating that type of surrogate for what you have done online...because that is what you will have: a different thing from your blog, but perhaps a wonderful thing in its own right.

And, you may wish to consider the fact that future generations may not have the same attachment to physical objects that you do. Printing out your blog *means* something and changes it fundamentally. This new thing definitely won't have the same qualities to you as it will to your grandchildren. This has always been the case, though, even with physical stuff.

In the end, I guess what I'm saying is that you may not be able to save your blog in the same way that your grandmother's diaries might have been saved. That doesn't mean you can't pass this wonderful resource down, though. You just won't be able to do it in the same way.

[Convert Slides To Digital]

Great information for digital storage, thanks for writing this for us to be informed too.

[External Hard Drives]

"The longer you leave your digital files alone without looking at them, the greater the risk that the information will not be useable when you go back to find it."I got it.

[ugg outlet store]

Even if the individual develops the same symptoms or health conditions that their parents or other family members had...chances are, if it is a degenerative condition...

[supra shoes]

Nothing is more precious than independence and freedom.

[coach outlet]

nothing is impossible for a willing heart.