The business requirements for record archives have evolved extremely rapidly over the last few years. Major financial scandals and a number of recent incidents involving large-scale data loss have turned the spotlight on the management of digital archives. An increased awareness of the value and liability of archive records has resulted in both industry regulation and internal operational risk management.
Today, organisations of all sizes and across all industries are subject to a wide range of local and international government and industry regulations. The most well known regulations seek to control financial records, legal information, health and safety data and access to personal or public records. In some cases these regulations can be very detailed, spelling out key criteria and procedures that must be met to ensure compliance.
Typical regulation criteria
Record types to be retained (data classes)
The retention period for each data class
Best practice and storage technology for legal authenticity
The final disposition or destruction of expired records.
If there was a single set of unified regulations, compliance might not be very difficult. Unfortunately, this isnt the case. Regulations within a country often conflict with each other and for those businesses operating internationally, foreign regulations add an entirely new dimension to the problem.
This complexity has forced many larger organisations to create the role of Compliance Officer or Risk Management Specialist. The task for this person is to assess the burden of external regulation against the organisations own internal risk management priorities and to set guidelines that satisfy both. This is no small task and is not without its own risk. As a result, internal policies on archival storage are often set to exceed the parameters defined by the regulations, while at the same time there is a strong desire to destroy records when legally permitted, in order to reduce corporate exposure.
Balancing these demands means that the management of archive records is becoming more complex, requiring different service levels to meet record retention and disposition requirements while minimising legal liability. This creates a whole host of new IT challenges. If they are not already, the IT team must become part of the business process. Gone are the days when IT existed in isolation from the rest of the organisation. IT administration must understand business priorities as well as any other group within the organisation if they have any hope of addressing these important issues.
The requirement for tighter integration of archive policies within an IT infrastructure is creating a demand for more flexible strategies that can accommodate the new regulatory and risk management burden. This need for flexibility is particularly important in the choice of physical storage media since it will, in large part, determine the success of implemented policies. This can be illustrated by examining two common archive objectives that are tightly linked to storage media attributes: record authenticity and record disposition (destruction).
Record Authenticity
For many archived document types it is essential to establish and maintain a very high standard for record authenticity. This is the case for any record subject to legal scrutiny including financial, medical and corporate communications (including emails). This is so important that some regulations specifically call for the use of WORM (Write Once Read Many) storage technology as one means of establishing a clear audit trail to ensure that records have not been altered.
In recent years, storage vendors have developed specialised magnetic disk-based RAID archive products sometimes referred to as CAS (Content Addressable Storage) solutions. Most of these products have implemented Write Once functionality through a mix of software and/or firmware that emulates Write Once capabilities on rewritable magnetic media. CAS solutions are available from EMC, HP, IBM, NetApp and Sun, to name just a few.
Tape vendors have also responded to the demand for record authenticity by developing tape-based WORM emulation products that use firmware and physical tabs on the tape cartridge to prevent data from being overwritten. Vendors such as Sun (StorageTek) and IBM offer high-end WORM tape, in addition to more mainstream AIT, LTO and DLT WORM products.
Plasmons UDO (Ultra Density Optical) professional optical product offers True Write Once technology implemented at the physical media level. The recording surface of True Write Once UDO media allows files to be written, but the media itself cannot be physically erased or modified. This technology is significantly different than magnetic disk and tape emulation since the Write Once properties of UDO are inherent to the recording surface of the media and are not a function of software or firmware controls.
Disk or tape WORM emulation may be acceptable depending on the authenticity requirements of the organisation, but only optical media provides unquestioned physical authenticity and is named explicitly as a preferred archive media by some regulations. The selection of the storage media can play a critical role in establishing the admissibility of digital records in a court of law.
Record Disposition
The issue of digital record disposition is emerging as a major consideration for many archives. Exactly how and when data can be destroyed is governed by some regulations and is at the heart of operational risk management. An archive strategy must find a way to balance regulatory requirements to retain records and a corporate desire to destroy them for both practical and liability motives. Here too, the choice of storage media plays a key role.
Some regulations define retention periods that allow data to be deleted after expiration and some go further by actually mandating record destruction and specifying the nature of destruction. Detailed destruction specifications are most common with documents related to security or personal information and typically call for the physical obliteration of the data. In these cases, deleting pointers to files or deleting keys to encrypted files is not sufficient. The records must no longer be physically present on the storage media.
If archiving on a typical RAID system, a simple delete operation does not remove the data from the disk. The only way to physically destroy records is by repeatedly overwriting the targeted sectors with a patterned sequence to ensure no residual trace of the document remains on the media. Depending on the source of the recommendation, targeted sectors should be overwritten between 3 and 35 times. The US Department of Defence has an often-quoted specification for data shredding on magnetic disk media (DoD 5220.22-M). This type of operation is not a standard file system feature but has been implemented in some of the specialised CAS products in the context of a record retention policy.
The destruction of records in an archive using magnetic tape is particularly difficult. The sequential data format used in writing files combined with the physical wear and tear on the media makes individual file destruction impossible. While full tapes can be erased and re-used, discrete records cannot be physically destroyed without totally rewriting the media. For similar reasons, consumer CD and DVD optical formats suffer from the same limitations as magnetic tape. In both cases, references to archived data can be deleted, but the actual records remain on the media. If assured data destruction is a key archive requirement, the use of magnetic tape, CD or DVD could be extremely impractical.
By contrast, UDO offers a Compliant Write Once media format designed specifically for data disposition requirements. Compliant Write Once UDO operates like standard WORM media, but has the ability to physically destroy targeted files through the use of a special shred operation. This is a one-pass function that provides full verification and unlike the erase pass on magnetic disks, the shred procedure on UDO media leaves no residual traces of previously written files. Compliant Write Once UDO media enables record level retention management with an extremely high standard for physical record destruction.
Archive Attribute Summary
Record authenticity and disposition are just two of many possible storage attributes to be considered when designing an archive. Others include access performance, capacity, media longevity and Total Cost of Ownership. The priority of these attributes will vary between organisations and among record types within the same archive. Given these diverse demands, it is vital to have an operational understanding of external regulations, internal risk management and the physical storage technology. The rapidly evolving nature of todays record archives demand products and strategies that enable the greatest possible flexibility.
Archival Storage Attributes CAS RAID WORM Tape True Write Once Media No No Yes Yes Data Destruction Yes No No Yes Removable Media No Yes Yes Yes Professional Quality Yes Yes No Yes Media Longevity Low Medium High High Media Capacity Med/High High Low Medium Seek / Access Performance High Low Low Medium Total Cost of Ownership High Low Low LowDVD
UDO
Steve Tongish is director of marketing (EMEA) at Plasmon--a leader in professional data storage solutions, providing a complete line of UDO drives and media, optical libraries, and Raidtec RAID storage systems for NAS, SCSI and fibre channel environments. Customers include corporations, institutions and government agencies worldwide and are backed by a global service and support network.
Add a Comment
No messages on this article yet