It’s no secret that the digital universe is growing at an amazing rate. Digital cameras are ubiquitous, with file sizes increasing with each new camera release. With easy-to-use digital video capture that is fed to consumers via streaming and other modes of IP transmission, technologies that used to be restricted to a handful of savvy users are now being accessed by millions of people on a daily basis.

A recent white paper published by the market intelligence firm IDC references the “digital tsunami” of information, noting “…the amount of information created, captured, or replicated exceeded available storage for the first time in 2007. Not all

Neel Patil

information created and transmitted gets stored, but by 2011, almost half of the digital universe will not have a permanent home.” The report also notes that approximately 70 percent of the digital universe is created by individuals, but enterprises are responsible for the security, privacy, reliability, and compliance of 85 percent.

Technology managers at corporations, universities, and government offices need to deal with an ever-increasing amount of digital content, and are concerned not only with capturing and storing this content, but also with archiving, indexing, and searching the content. At schools and universities, most class content is now stored digitally, and many places are generating more content daily.

The University of California, San Diego, captures daily audio podcasts for classes with the Academic Computing Services department hosting the captured materials on their servers. Howard Laurence, UCSD media services manager, says that although podcasts do not occupy much space, the school is aware of the growing need for storage as they increasingly capture video and audio for special events. “Our department will be doing a bit more of this hosting in the future, as we are acquiring our own department-owned servers,” Laurence says.

In the corporate world, users need not only reliable data storage for

PHOTO COURTEST OF IBM

email, financial information, and electronic documents, but for the capture, storage, and retrieval of AV content as well. Videoconferences and training videos, streaming for employee updates, digital signage, etc., are rapidly gaining prominence in daily business. Storage of this content is of great concern to all users. But should AV content be stored in the same way as regular data?

THE AV DIFFERENCE
The short answer is “no” because AV content involves many types of data with linear temporal content. A typical file comprises a container (also called a “wrapper”) and varying content within that wrapper, according to Dave Frederick, senior director of market development at Omneon. He cites as an example: “A QuickTime file has a container with video, a stereo audio file, and possibly some closed captioning or metadata or other things contained in the wrapper. Being able to keep track of these pieces of information, move them together, know when it’s OK to break them apart, get into the wrapper and do things with it, and so on differentiates traditional data storage from AV media storage.” Frederick also notes that the challenge of storing media files is compounded by their size. “Professional AV content tends to be very large, frequently multiple tens of gigabytes per file,” he says. “The size itself creates problems with the ability to get it in and out of storage quickly and usably.”

Josef Marc, VP of customer engineering for Front Porch Digital, says there is another critical difference between storage of data and AV. “Data storage is primarily driven by financial industries where they have to keep a lot of transactions,” says Marc. “This type of IT storage is not normally optimized to keep things together; it will break things up randomly for storage efficiency, but that isn’t a problem in banking. It is in video.” Marc also notes that the temporal content of video and other types of media makes it necessary to keep everything in the file together, which can make AV storage a bit trickier than data.

Marc cites LTO tape (linear tape-open), which was developed for storage of standard data files, as a good option for long-term AV storage, but has a caveat. “You need to be careful about how the media are fed to LTO,” says Marc. “Data files are generally no problem, and if the tape needs to stop and back up and go again, that’s ok. But if you do that a lot, you get a shoeshine effect on the tape. LTO is great for copying videotapes for storage, but if they are not fed to the LTO just right, the LTO will keep backing up and will cause a shoeshine effect that wears out the tape more quickly.”

Some users find LTO a convenient method of storing AV content, and Front Porch Digital uses it extensively. Marc says, “Hard drive costs are coming down in terms of cost per bit, but the power to keep them on is an issue, so tape storage is still of value —especially with energy costs going up and other energy concerns.”

Omneon’s Frederick says LTO is a viable option for some users, but says, “If you want media access to entire libraries with latency of no more than a second, you won’t achieve that with an LTO system or even videotape. You’ll need a spinning disk with multiple heads, rate arrays, flash media, solid state, or something similar.” Ultimately, the choice of storage media depends on what you need to do with your media, how often you need to access the files, and how quickly you need to access them. However, most experts agree that the storage capacity you’ll need for AV is much higher than the amount you’ve been using for data.

Is online, or “cloud,” storage a viable option? If you are dealing with tens of gigabytes at a time and your connection to the cloud is a DSL line, it could take a very long time to upload a file, and could take an equally long time to retrieve it. For some users, however, offsite online storage is a good option because it allows them to put their media in the hands of people who professionally manage it and create customized archiving and search tools for the media. However, some users such as government agencies may not have that option due to the sensitive nature of the media they are storing — it may actually be illegal to move certain files offsite.

Companies such as Front Porch Digital specialize in assisting customers with migrating old media, such as video tapes, to digital formats. “It’s important to preserve old media and transfer it to new storage media before the tapes rot and become unusable,” says Marc. He notes that some older videotape machines, such as U-Matic, are becoming very difficult to find these days; there are very few decks available and very few parts on the market. “Our company may be contributing to this shortage because we buy them as fast as we can find them to assist our customers with migration to new storage media,“ Marc says.

EASY ACCESS
Once you have placed media content into storage, how do you access it? Frederick notes that simply pulling files across the network can be problematic. “For example, you have a file in your central library and someone wants a version that incorporates the middle three minutes into a new edit,” he says. “In typical storage you would copy it to your local editing client, make your edits, finish the project, and put it back. During the time that these files are being transferred, it chews up network availability for others, and it also creates a time lag in when you can start your work and finish it. That’s not an inconsequential amount of time — up until very recently, it was faster to drive tapes across town than put them on a network.” Companies such as Omneon and Front Porch Digital specialize in developing infrastructure and storage platforms that optimize the workflow when dealing with digital AV content.

Both Frederick and Marc stress the importance of correctly tagging and archiving the various versions of a file that could exist in your storage system. For example, a digital signage application may involve high-resolution video that is shot and stored on your media server. This content is then accessed by an editor who creates a custom file that is to be sent to 1,500 stores. This new, custom file is lower resolution than the original, but contains similar content and needs to be tagged with not only its content, but its intended use to properly differentiate it from the original.

Karen Krier of the Utah Education Network, based at the University of Utah, says asset management is critical to accessing content at the school. “Distance learning instructors tend to store their own learning materials for each class, but we have a library of items that are used by both distance learning and by on-campus instructors,” Krier says. “We are storing close to 13,000 assets at the U of U, including images, video, PDF files, audio, and so on. We use an external asset management system that allows teachers to search for a video, image, or other file that would be helpful for the class, then the teacher downloads that content for each class.” UEN relies on titles and metadata to archive and search content; Krier says creating the metadata for each file is time consuming, but absolutely essential.

Although most video today is keyword metadata searchable, Frederick says asset tagging is becoming more advanced, and future users will have some very useful tools at their disposal. “Some companies are developing systems that build in facial recognition as part of the tagging,” he says. “With this type of system, if you did a search for Woody Allen, it could look through the library and find instances of him being on screen. Other companies are working on turning audio into searchable text and tracking that text against timecode. You will also be able to investigate closed captioning information along with metadata.”

What about long-term storage — keeping items for posterity? “I think everyone’s afraid to talk about that,” says Frederick. “Even CDs are beginning to become suspect; many CDs from the 1980s are not reading correctly now. There are going to be new formats, new devices, and, as you create larger and larger libraries, any comprehensive storage plan will have to include migration of that content across generations of storage. That’s going to be a challenge.”

A long-term plan for media storage is essential, but there are now many resources available to help you deal with that challenge.

Elaine Jones operates a PR agency specializing in the AV industry and is based in Salt Lake City, UT. She can be reached at elaine@ejonespr.com.