The Storage Architectural Landscape

Storage Building Blocks
The basic elements of a computing storage system haven't changed in the last 30 or so years. The equipment and areas that control the primary interaction may have moved around, but at the end of the day, an operating system still initiates a request to a file system, which in turns talks to a volume management layer, and ultimately the storage devices themselves, as shown in Figure 2-1.

Figure 2-1. Basic storage building blocks.


The earliest server implementations of this architecture took place internal to the system but have since expanded outside the box into specialized equipment and software platforms. Intelligent disk arrays, switching infrastructure, and network transports have shattered the traditional blueprint for storage designs. This newfound geographic freedom for a wide variety of storage functions presents IT professionals with previously unavailable architectural options. In today's global computing environments, these options help save time and money.

The SNIA shared storage model serves as an excellent reference for outlining storage architectures. Specifically, the model provides a navigable map for the variety of functions that make up a complete storage system.

2.1.1 STORAGE GUIDANCE: File Systems, Volumes, and Blocks
The model in Figure 2-2 provides an abstraction of how applications address storage, allowing IT administrators to effectively plan complete architectures. It carries the simplified model in Figure 2-1 further by calling out specific mechanisms that file systems and volume management use to access storage devices. In the case of the SNIA model, one can roughly equate volume management with block aggregation.

Figure 2-2. The SNIA shared storage model.


Applications are the common starting point for the model. From there, all other mechanisms fall into the storage domain—database management systems (DBMS), file systems (FS), and all of the ways to connect to disks. The connection to the physical disk drive may be direct from the server or host, through a storage network, or through a device such as a NAS head or NAS server. Each of these areas—host, network, and device—may perform the block aggregation functions required to deliver blocks to the upper-layer file or record layer.

One benefit of the SNIA shared storage model is the ability to map a variety of storage architectures onto a single framework. In the case of large corporations with storage deployments spanning direct-attached storage (DAS), NAS, and SAN-attached storage, viewing these seemingly separate hardware infrastructure across common dimensions can help rationalize purchases and resources across the storage organization. Further, this optimized view helps IT managers plan and architect corporatewide solutions serving multiple departments, enabling greater efficiency.

Figure 2-3 highlights the versatility of the model in depicting multiple architectures within one framework. Item one, DAS, connects directly from a host, or server, with its own logical volume management (LVM) to a set of disks. In this case, the host provides software redundant array of independent disks (RAID) functionality to the unintelligent disks, often referred to as just a bunch of disks (JBOD). File system and volume management functions reside on the host. Note that the host box covers the entire space of the file/record layer, and covers host block aggregation in the block layer of the diagram. Item two is similar, but now the host connects to a disk array across a SAN. It still has some host block aggregation (i.e., volume management), but now some of the aggregation, such as the RAID functions, are shared by the device, which overlaps with device block aggregation in the block layer of the diagram.

Figure 2-3. The SNIA shared storage model depicting direct-attached, SAN-attached, and network-attached storage.


Items three and four change the storage access mechanisms to file-based, as opposed to block-based. Note that a local area network (LAN) resides between the host and the NAS head and NAS server, and that each NAS unit overlaps into the file/record layer. The communication requests from the hosts to the NAS units are file-based, used for corporate file sharing, for example, compared to block-based hosts (items one and two), which use a SCSI-based access mechanism (SCSI, Fibre Channel, iSCSI) to communicate with the disks.
Internal Storage/Server-based storage
In early computing designs, storage devices were located within the computing system itself, such as a server. In this case, all storage components (file system, volume management, and storage devices) are internal. This simple implementation is easy for administrators to deploy, with each server having specific storage allocations met by internal disk drives.

Disk drive interfaces vary between SCSI (Small Computer System Interface), IDE (Integrated Device Electronics), and ATA (Advanced Technology Attachment), with each providing a cost/performance optimization suited to varying market segments. Inside an enclosure, the disk drive interface has little effect on the overall enterprise computing system design. But placing individual storage devices within each server leads to scalability issues. In an internal-only storage environment, each application is largely constrained by the amount of storage that can fit inside the box. Beyond that, the only options left are moving outside the server.
External Storage (JBOD, RAID, Tape, Optical, Other)
External to the server, storage devices have the flexibility to evolve to highly specific task-oriented machines, from JBODs at the low end of the disk-based media spectrum to high-end RAID systems. Similarly, tape, optical, solid-state, and other forms of storage come in all shapes and sizes to suit specific needs. This section covers the basics of external storage devices leading up to transition from direct-attached to networked storage.

An understanding of these basic storage components provides significant value as IT professionals begin to chart their storage networking roadmaps. Since external storage devices vary in size, cost, and complexity, effective use of these basic building blocks helps offset potential pitfalls as the entire storage infrastructure scales.

2.3.1 Just a Bunch of Disks (JBOD)
While some may place JBODs at the low end of the storage food chain, a careful look at the overall storage networking picture reveals more sophisticated uses for these devices. With a relatively low cost per megabyte for disk-based storage, JBODs provide excellent value where redundancy and intelligence can be added elsewhere in the overall architecture, for example within a host or network.

The basic JBOD is an enclosure for 3.5-inch hard disk drives that provides a daisy-chain mechanism to link the devices together. In SCSI terms, the disk drives are independent devices on a parallel SCSI bus. In Fibre Channel terms, the disk drives are independent devices on a Fibre Channel loop. A typical JBOD architecture based on Fibre Channel loop is outlined in Figure 2-4.

Figure 2-4. Basic JBOD design.


2.3.2 Redundant Array of Independent/Inexpensive Disks (RAID)
RAID can be implemented directly in storage hardware devices or in storage controllers, such as host bus adapters (HBAs), or in software. While these options offer choice in terms of cost and performance, the basic principles of RAID remain the same. RAID architectures protect disk drive-based data with several redundancy mechanisms to protect against disk failure. Implemented properly, storage systems using RAID will be unaffected by disk drive loss and can go uninterrupted through the cycle of disk failure and repair.

RAID grew considerably with the introduction of inexpensive 5.25-inch and 3.5-inch drives. Manufacturers valued the opportunity to use less expensive components with a cost curve similar to the PC industry, but also had to contend with the potential failure of such low-end components. The specific terminology used to describe the life of a disk drive is mean time between failure (MTBF). By removing individual disk drive failure from the overall system MTBF, RAID systems could guarantee higher availability and continuous uptime for users.

Today, high-end RAID systems offer a variety of redundancy measures and storage management features that contend directly with similar offerings in the rest of the enterprise. A mirror, or exact copy of the data on an additional set of disk drives, can operate directly within a RAID system for optimized performance. Similar functions can also be accomplished directly with host-based software or within a storage network.

The primary RAID functions include striping, mirroring, and parity. IT professionals should be familiar with these functions and terminology, as outlined next in Section 2.3.3.

2.3.3 STORAGE GUIDANCE: RAID Levels
Basic RAID features break down into a series of defined RAID levels as originally documented by David A. Patterson, Garth A. Gibson, and Randy H. Katz of the University of California at Berkeley in 1988. Six basic levels were defined—RAID 0 through 5 (see Figure 2-5). Each represents a unique mechanism to achieve higher performance and availability with common disk drive components. The numbers do not directly correlate with degrees of performance or availability and should be viewed categorically. In practice, RAID 0, 1, and 5 are the most commonly implemented RAID mechanisms. RAID 2 required special disk features. Given the economics of volume production, the cost of such features favored simply using other RAID levels. RAID 3 and 4 also are not frequently implemented due to the performance impact of dedicated parity disks.

Figure 2-5. RAID levels.


RAID levels, outlined in Figure 2-5, can be combined to create even higher levels of availability. For example, a RAID 0 array may also be mirrored. Known as RAID 0+1, this provides the performance of RAID 0 striping with the redundancy of RAID 1 mirroring. Another configuration, RAID 10, stripes data across two independent mirrored disk volumes. Similar combinations exist with RAID 0 and RAID 5.

2.3.4 Tape Drives and Tape Libraries
Purchasing decisions for storage devices measure performance, features, capacity, and cost. At the high end of the performance, feature, and cost equation are large RAID arrays. While it would be convenient for enterprises to standardize on a single set of media devices, economics come into play and create opportunities for other device types.

Tape drives and tape libraries offer high-capacity, low-cost storage when compared to disk arrays. Tape density for single tape drives and cartridge density for automated tape libraries far outweigh the equivalent capacity of a disk-based unit with a similar footprint. Further, the ability to swap tape cartridges provides flexibility to add capacity and the ability to remotely locate tapes for disaster recovery purposes.

Today, SANs allow consolidated sharing of tape libraries across a number of servers. Whereas previously, each server required its own tape drive, or servers had to share traffic on the LAN for access to a central backup server, SANs allow direct connections between servers and tape libraries for simplified backup, offloading a potentially overburdened LAN. Coupled with the appropriate backup software package, this provides an economical, easily managed backup system.

Tape media comes in a variety of forms, such as digital linear tape (DLT), linear tape-open (LTO), and advanced intelligent tape (AIT), each offering its own capacity, performance, and cost equation that will manifest itself in the overall system specifications. Choice of a tape library should focus more on system features within the context of the enterprise storage architecture rather than choice of a specific media type. However, purchasers should be advised that media types do have a life of their own. Yearly checks to assure both media type and subsystem availability should be part of data center operating procedures. To protect against outdated media types and potential data loss due to tape degradation or magnetic interference, IT managers often re-record data to fresh tapes every year. This guarantees recovery mechanisms but also adds the cost of tape replenishment.

2.3.5 Other Storage (Optical, Solid State)
Other media technologies include optical, which is often compared to tape, but since it uses electromagnetic recording mechanisms, the shelf life of the discs far exceeds that of tape. Electromagnetic mechanisms for optical recording operate at much higher temperatures than traditional magnetic disk and tape recording. This prevents interference from nearby magnetic material or environmental impacts due to low or high temperatures. In fact, if kept in a properly controlled environment, optical media suffers virtually no degradation over time.

Optical media comes in recordable CD/DVD form, 5.25-inch WORM (write once, read many), and 12-inch optical disks. These magnetic-optic (MO) drives operate independently or can be assembled in to a library (often called a jukebox) to provide high-capacity optical storage similar to tape libraries. In an automated jukebox, mechanics allow for recording on both sides of the optical disk.

Solid-state storage media sits highest on both the performance measure and cost measure. It is often used for performance-critical applications that operate from a small data set. As the name implies, solid-state storage relies upon memory chips to provide capacity. This implementation provides extremely high throughput and I/O operations per second. However, due to cost and capacity limitations, solid-state storage is used in limited market segments.

Direct-Attached Storage
DAS expands server storage capacity. It provides easily configurable storage that looks local, or internal, to a server, yet has scalable capacity benefits of being outside the server itself. External storage can extend server life by providing not only additional capacity but a range of storage features, such as redundancy, that help protect data.

With DAS, servers access blocks of data. The file system and volume management layers are host-based. Other data storage functions implemented independent of the server, such as RAID, provide greater performance and leave the server free to focus on its priority tasks, such as application processing. Most external storage devices attach to servers via high speed parallel SCSI or Fibre Channel cabling. Traditionally, the speed difference between SCSI and Fibre Channel compared to LANs has been significant, although today's high-speed networking interconnects, such as Gigabit Ethernet, level the playing field. See Section 2.6.3, "SAN GUIDANCE: Comparing Ethernet/IP and Fibre Channel Fabrics," for a more complete overview of storage and networking interconnects.

However, even DAS has its limits. Adding additional storage usually requires incremental servers, and one server does not have direct access to another server's storage. This captive storage model makes resource sharing difficult and leaves servers susceptible to single points of failure. The basic DAS model is shown in Figure 2-6.

Figure 2-6. Direct-attached storage.


Reaching the limits of the captive storage model, solutions with additional scalability arose that networked storage together. This development took shape in the mid-1980s, well before the introduction of high-speed serial storage technologies such as Fibre Channel and iSCSI. Therefore, without a means to serialize the high-speed parallel SCSI interconnects between servers and DAS devices, the only other available network was the Ethernet and IP-based LAN. This led to the introduction of NAS.
Network-Attached Storage
NAS devices overcame several barriers when introduced to the storage market. First, NAS broke the DAS captive model. Second, it provided a mechanism to link storage across Ethernet and IP networks, a completely new footprint for storage administrators. Most importantly, and in part to support the architectural change, NAS moved the file system and volume management functions from the server to a NAS device, often called a filer. Keep in mind that the "filing" function remains no matter what the storage model. The location of this function has traditionally separated the distinction of NAS and SAN. See the following section for a more complete outline of the differences between NAS and SAN.

2.5.1 STORAGE GUIDANCE: SAN and NAS
A discussion about accessing storage via IP and Ethernet, such as with IP storage adapters, invariably begs the question about the difference between SAN and NAS. The SNIA shared storage model outlines file-level and block-level distinctions, which are further clarified in Figure 2-7.

Figure 2-7. Comparing SAN and NAS.


SANs provide scalable performance with direct access to storage devices. Since the storage system resides on the distributed hosts, this architecture works well for databases and online transaction processing.

NAS provides crossplatform scalability, but not the performance scalability due to its use of a high level of file abstraction between applications and storage. While a NAS filer offloads some server workload required to map files into physical storage, it adds significant workload to the filer itself. NAS delivers a central storage system (file system) that is ideal for file sharing, Web serving, and similar applications.

A NAS device is essentially a highly optimized file-delivery server. Depending on the specific implementation, NAS devices may have internal storage (similar to a traditional server), or they may be directly connected to a RAID device for additional capacity or to a tape library for backup functions. Additionally, some NAS devices can connect directly to a block-based storage network.

Traditional servers access NAS devices through the Ethernet and IP network by means of a file request. Since the file system resides on the NAS device, servers with different operating systems can all easily access the same data. This crossplatform access gives NAS a considerable advantage in terms of simplicity. It also allows the NAS device to provide lock functions for particular files. Since the device knows if one server is accessing a file, the second server will not be able to write or modify that file.

NAS has made tremendous inroads to certain market segments where the architecture suits the applications. Web-related functions, such as serving HTTP requests, are ideal applications. The relatively small file size combined with the crossplatform access across Ethernet and IP networks makes NAS a winner.

Similarly, corporate file sharing is easily addressed by NAS. To add more storage capacity to such a configuration, a storage administrator would simply need to plug another NAS device into the IP network. Of course, if for the new NAS device to be completely integrated with other NAS devices in a network—for example, where the NAS devices could pool storage capacity—purchasing the unit from the same vendor becomes a requirement.

Figure 2-8 outlines a basic NAS architecture. As shown, application servers access NAS filers over the IP network, providing greater scalability and crossplatform access than DAS provides. However, NAS doesn't alleviate LAN congestion, and the expansion of block-oriented storage remains disruptive. Additionally, each filer owns the storage that it maintains (except in some homogeneous, pooled NAS environments), and in general, NAS has a unique OS that must be maintained across all filers for ease of management. Finally, the high level of file abstraction inherent to NAS design doesn't suit certain application requirements accustomed to block-level storage.

Figure 2-8. Network-attached storage.


The gap between high-performance DAS and flexible NAS became the sweet spot for SANs. Traditionally, block-oriented SANs solved a combination of problems between DAS and NAS with Fibre Channel networks. More recently, SANs have been able to fill this gap with block-oriented IP SANs, using IP storage protocols such as iSCSI. The following sections cover the details of both Fibre Channel and IP SAN infrastructure.