CA2063897C - Method and means for distributed sparing in dasd arrays - Google Patents
Method and means for distributed sparing in dasd arraysInfo
- Publication number
- CA2063897C CA2063897C CA002063897A CA2063897A CA2063897C CA 2063897 C CA2063897 C CA 2063897C CA 002063897 A CA002063897 A CA 002063897A CA 2063897 A CA2063897 A CA 2063897A CA 2063897 C CA2063897 C CA 2063897C
- Authority
- CA
- Canada
- Prior art keywords
- blocks
- dasd
- parity
- dasds
- array
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000003491 array Methods 0.000 title claims abstract description 47
- 238000000034 method Methods 0.000 title claims abstract description 39
- 230000001360 synchronised effect Effects 0.000 claims description 31
- 125000004122 cyclic group Chemical group 0.000 claims 4
- 238000009826 distribution Methods 0.000 abstract description 14
- 238000011084 recovery Methods 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000006243 chemical reaction Methods 0.000 description 2
- 230000007423 decrease Effects 0.000 description 2
- 230000036039 immunity Effects 0.000 description 2
- 230000033001 locomotion Effects 0.000 description 2
- 230000008569 process Effects 0.000 description 2
- 230000007480 spreading Effects 0.000 description 2
- 230000004931 aggregating effect Effects 0.000 description 1
- 230000002860 competitive effect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 238000009827 uniform distribution Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
- G06F11/2094—Redundant storage or storage space
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1076—Parity data used in redundant arrays of independent storages, e.g. in RAID systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/1658—Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit
- G06F11/1662—Data re-synchronization of a redundant component, or initial sync of replacement, additional or spare unit the resynchronized component or unit being a persistent storage device
-
- G—PHYSICS
- G11—INFORMATION STORAGE
- G11C—STATIC STORES
- G11C29/00—Checking stores for correct operation ; Subsequent repair; Testing stores during standby or offline operation
- G11C29/70—Masking faults in memories by using spares or by reconfiguring
- G11C29/88—Masking faults in memories by using spares or by reconfiguring with partially good memories
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2211/00—Indexing scheme relating to details of data-processing equipment not covered by groups G06F3/00 - G06F13/00
- G06F2211/10—Indexing scheme relating to G06F11/10
- G06F2211/1002—Indexing scheme relating to G06F11/1076
- G06F2211/1028—Distributed, i.e. distributed RAID systems with parity
Abstract
A method and means in which data and parity blocks forming parity groups together with spare blocks are distributed over array block locations according to at least one combinatorial design, each group having N data and P parity blocks. The combinatorial designs yield uniform or balanced loading thereby minimizing the number of accesses to reconstruct missing data and parity blocks and their copyback into spare block locations, and, minimize the number of accesses to the reconstructed data referenced subsequent to its copyback. Distributions of the spare block capacity of one or two DASDs are shown over single and multiple arrays and shared among multiple arrays. Parity block distribution although ancillary to spare distribution enhances throughput and reduces the number of accesses for rebuild etc.
Description
CA 02063897 1997-0~-28 METHOD AND MEANS FOR DISTRIBUTED SPARING IN DASD ARRAYS
Field of the Invention This invention relates to DASD (Direct Access Storage Device) storage subsystems, and more particularly, to methods and means for managing spare DASD array capacity so as to optimize array operations in fault tolerant, degraded, and data rebuild modes.
Description of Related Art Arrays, Effect of Redundancy, Reading and Writing In the prior art, it is lcnown to read and write data + parity (as defined over the data) from and to a synchronous array of N data + P parity DASDs. The DASD
array arrangement increases the data rate by N * rate of a single DASD and increases logical track size by N * single DASD track length. Reference can be made to Patterson et al, "A Case For Redundant Arrays Of Inexpensive Dislcs (RAID)", Report No. UCB/CSD 87/391, December 1987, Computer Science Division, U. of California, Berlceley.
It is also lcnown that writing a data string to a DASD array includes segmenting data into N bloclcs (terrned striping), determining parity over the bloclcs, and storing the N data + P parity bloclcs at a synchronous address on counterpart failure independent DASDs. Likewise, reading a data string from an array involves copying N+P addressed bloclcs from a synchronous address on counterpart DASDs into a buffer, concatenating them, checldng parity, and serially transporting the concatenated blocks (string) from the buffer to the accessing CPU.
DASD Failure and MTBF
It is well recognized that aggregating DASDs into arrays does not improve the mean time between DASD failure. Rather, it decreases it. However, the combined CA 02063897 1997-0~-28 use of redundant information (parity), dynamic substitution of formatted spare DASDs for failed ones, and reconstruction of missing data onto the substituted spare, substantially increases the mean time between data unavailability by at least orders of magnitude. This is described in Park et al, "Providing Fault Tolerance In Parallel Secondary Storage Systems", Princeton University Report CS-TR-057-86, November 1986 and Dunphy et al, US
Pat. 4,914,656, "Disk Drive Memory", issued April 3, 1990 .
Two Usages of Parity Groups The term "parity group" has acquired data oriented and ~torage oriented usages.
In the data usage, "parity group" signifies a predetermined number of logically associated data blocks + a parity or equivalent redundant information defined over or derived from the data blocks. In the storage usage, "parity group" signifies a predetermined number of logically associated physical storage locations whose data contents determine the value of the redundant information.
Parity Spreading and DASD Arrays One method for managing DASD array accesses when operating in degraded mode, uses parity group as a logical association of data blocks as applied to Patterson s RAID 5 type DASD array. The data and storage boundaries of the parity group are not necessarily coextensive. The only limitation is that no two segmented blocks from the same data parity group be written onto the same DASD.
In contrast, Dunphy et al, US Pat. 4,914,656, "Disk Drive Memory", issued April 3, 1990, defines parity over data as in the above method but the storage boundaries are the same as that of the data parity group. They are maintained even in the presence of failure since a spare DASD substitutes for the failed DASD and missing data is rebuilt and rewritten onto the substituted spare.
CA 02063897 1997-0~-28 9_90_079 3 2063897 Clark et al, US Pat. 4,761,785, "Parity Spreading Enhanced Storage Açcess", issued August 2, 1988 discloses a non-synchronous DASD storage subsystem in which parity groups are defined over a range of storage and where data boundaries are NOT necessarily coextensive with storage boundaries.
In Clark et al, each data string is written into consecutive locations of a "storage" parity group. If the data string size exceeds the capacity of the group, then the residue of the data string is written into another "storage" parity group. If the data string is smaller than the group, then the remaining space may well be occupied by other data strings. Parity is taken across information occupying logically associated extents (range of address locations) of the DASDs forming the group.
Distributed Parity and Virtual or Distributed Sparing Clark et al also taught that the location of the parity blocks for counterpart storage parity groups could be distributed across DASDs in the subsystem with the only limitation that not all of the blocks be written on a single DASD. This was contrary to Ouchi, US Pat.
4,092,732, "System for Recovering Data Stored In A Failed Memory Unit", issued May 30, 1978, and Dunphy et al where parity is written to dedicated DASDs.
Another procedure for recovering parity protected data teaches the use of a virtual spare among non-synchronous DASDs where parity groups are defined across storage, and data and storage boundaries are not necessarily coextensive.
In this last-mentioned procedure, the CPU can read and write to a logical spare DASD. The logical addresses are then distributed in non-specific manner across one or more real DASDs. More particularly, it teaches that the CA 02063897 1997-0~-28 locations of parity blocks distributed as in Clark may be overwritten as if they were spare locations in the reconstruction of data being accessed after a DASD has failed. Usually, this means the parity block of the parity group covering the lost data.
In such a type of distributed sparing via writing reconstructed data into the group parity location, once recovery is complete, the system operates without parity.
This is inimical to a highly available, fault-tolerant system.
However, what this last-mentioned procedure failed to recognize was that:
a. two or more data blocks from the same storage parity group are written on the same DASD. If a DASD were to fail now, the DASD array subsystem would lose data, which is unacceptable.
b. data blocks of the same group formerly located on different DASDs now being resident upon the same DASD
cannot be read in parallel.
c. lowered throughput because of resource sharing among competitive processes occasioned by DASD arm stealing between the reading of the surviving data blocks to compute any missing data or parity block and the writing of a reconstructed data or parity block.
Summary of the Invention It is an object of this invention to devise a method and means for facilitating reconstruction of missing data and parity blocks and copying them back into spare block DASD locations.
It is a related object to devise a method and means in which the data and parity blocks forming parity groups together with spare blocks are distributed over array block locations according to at least one combinatorial design, each group having N data and P parity blocks.
Additionally, it is yet another object that said combinatorial designs minimize the number of accesses to reconstruct missing data and parity blocks and their copyback into spare block locations.
CA 02063897 1997-0~-28 It yet another object that the throughput be maximized during reconstruction and copyback of missing data or parity blocks and subsequent reference thereto.
Further, said method and means should be readily extensible to storage subsystems addressing two or more failure independent DASD arrays. Also, said method and means should be extensible such that the combinatorial design should be distributable over multiple failure independent arrays and with regard to different sparing ratios or fractions.
The foregoing objects are satisfied by a method and means for rebuilding portions of parity groups resident on a failed DASD, in a storage subsystem having a plurality of DASDS and where each DASD stores K blocks.
Also, each parity group includes N data, P parity, and S
spare blocks.
The method and means include configuring an array of N+P+S DASDs. Next, K parity groups are distributed in synchronous array addresses across subsets of the N+P+S
DASDs such that no two blocks from the same parity group reside on the same DASD. K*S blocks constituting spare storage are distributed across the array such that each DASD includes K*S/(N+P+S) spare blocks thereon.
Furthermore, no more than S spare storage blocks nor more than P parity blocks are stored on the same synchronous array address or on the same DASD.
In the event of a single DASD failure, for each of the (K-(K*S/(N+P+S))) affected parity groups, N+P+S-2 blocks belonging to the group are logically combined (XOR ed) from the other N+P+S-2 other DASDs into a single block. Each single block is written into a counterpart one of the remaining K*S*(N+P+S-l)/(N+P+S) spare blocks such that no two blocks of the same parity group are distributed on the same DASD.
In a first embodiment (where P=l, S=l), the method and means include configuring an array of N+2 DASDs. The next step involves distributing K-K/(N+2) parity groups in synchronous array addresses across subsets of N+1 DASDs of the array such that no two blocks from the same parity group reside on the same DASD. Concurrently, K
CA 02063897 1997-0~-28 '9-90-079 6 2 0 6 3 8 9 7 blocks as spare storage are distributed across the array such that each DASD includes K/(N+2) spare blocks thereon. As in the general case, no more than one spare storage block nor more than one parity block are stored on the same synchronous array address or on the same DASD.
In the event of a single DASD failure, for each K-K/(N+2) parity groups, N blocks belonging to the group are logically combined from N other DASDs into a single block. Each single block is written into a counterpart one of the remaining K*(N+1)/(N+2) spare blocks such that no two blocks of the same parity group are distributed on the same DASD.
In another embodiment (where P=l, S=2), 2*K blocks of spare storage and (K-2*K/(N+3)) parity blocks are distributed such that no more than two spare storage blocks nor more than one parity block are stored on the same synchronous array address nor on the same DASD. This permits rebuilding and writing missing data to a first series of spare blocks after a first DASD has failed and repeating the process in the rarer event that a second DASD should fail.
The method and apparatus of this invention are extensible to storage subsystems having multiple failure independent DASD arrays. Where multiple DASD failures occur in the same array, the missing data is first rebuilt on the spare space of the first array and the remaining missing data rebuilt on the spare space of the second array.
Where storage subsystem expansion involves one array with distributed spare capacity and an array without a spare, such capacity may be conveniently shared. Also, the blocks representing the capacity of one or more spare DASDs can be distributed across multiple arrays so that no synchronous address or DASD has more than one such spare block increment. Next, the parity groups can be written across their respective arrays in a rotated or block offset manner. This would permit a uniform sharing.
Other distributions of parity groups and sparing are disclosed which permit storage subsystem DASD array CA 02063897 1997-0~-28 ~9-90-079 7 2063897 expansion while maintaining a sparing fraction or ratio objective.
Brief Description of the Drawing Figure 1 shows parity groups coextensive with storage bounds using dedicated parity and spare DASDs and reconstruction of missing data or parity onto the spare DASD according to the prior art.
Figure 2 depicts distributing K parity groups {(4+P)<(N+2)} and K spare spaces over an array of N+2 DASDs of capacity K blocks or spaces/DASD permitting recovery from a single DASD failure according to the invention, no two elements of the same group nor space being located on the same DASD.
Figure 3 also sets out a distribution of K parity groups {(3+P)~(N+2)} and 2K spare spaces over an array of N+2 DASDs of capacity K blocks/DASD permitting recovery of up to two DASD failures according to the invention, no two elements of the same group nor spare space being located on the same DASD.
Figure 4 illustrates two DASD arrays with distributed sparing.
Figures 5 and 6 show failure of one or more DASDs in a first one of two arrays and the rebuilding of missing data on the distributed spare spaces across both arrays.
Figure 7 depicts partially distributed sparing.
Figure 8 depicts a synchronous array of N+2 DASDs attached to a CPU by way of an array control unit.
Despcription of the Preferred Embodiment DASD Array Architecture Referring now to figure 8, there is shown CPU 1 accessing DASDs 1 through N+2 over a path including channel 3, array controller 5, and cache 13. Controller 5 operatively secures synchronism and accesses among any N+1 at a time of the N+2 DASDs i.e. DASD 1 through DASD
N+2 over control path 7.
N+1 streams of data defining a predetermined number of consecutive bytes can be exchanged in parallel to CA 02063897 1997-0~-28 ~9-90-079 8 cache 13 over data path 15. The N+l streams of data may all be responsive to a single access (synchronous operation). Also, this invention is operative where each of the N+l streams may be responsive different accesses (asynchronous operations). Likewise, data can be exchanged serially by byte between CPU 1 and controller 5 over path 3 after a parallel to serial conversion in controller 5 in the read direction and a serial to parallel conversion in the write direction.
In the read direction, data is supplied from cache 13 to controller 5 via data paths 9 and 11. In the write direction, data is moved from the controller 5 to the cache 3 over paths 9 and 11.
Parity, DASD Failure and Sparing, and Robustness DASD arrays use parity to protect against single DASD failures. If a DASD fails, data that used to be on that DASD can be reconstructed, as needed, using the data and parity on the surviving DASDs. This is illustrated in Table 1 in a DASD array of five DASDs. In this diagram, Pi is a parity block that protects the four data blocks labelled Di. Such a DASD array is called a 4+P array, since there is one parity block for every four data blocks.
DASD 3 Dl D2 P3 D4 D5 Table 1.
Only one track (consisting of five blocks) is shown from each of the DASDs. Pl contains the parity or exclusive OR of the blocks labeled Dl on all the data DASDs. Similarly, P2 is the exclusive OR of the blocks labeled D2 on all the DASDs, and so on. It should be noted that the parity blocks are distributed amongst all the drives to balance the workload.
Such a DASD array is robust against single DASD
failures. If DASD 1 were to fail, data on it can be recreated by reading data and parity from the remaining four DASDs and performing the appropriate exclusive OR
operations.
An array is said to enter "degraded mode" when a DASD in the array fails, because the performance and reliability of the array becomes degraded. Performance is degraded since every DASD access to a block on the failed DASD now requires DASD accesses to other DASDs in the array for reconstructing the block that is no longer accessible. Reliability is degraded, since if a second DASD fails before the failed DASD is replaced and the data on it reconstructed, the array will lose data.
In the specification of this invention, the term "reliability" indicates the degree of immunity from data loss it possesses. The higher the immunity from data loss, or the higher the mean time to data loss (MTTDL), the higher the reliability of the array.
To minimize the probability of losing data and the length of time the array operates with degraded performance, arrays sometimes use "hot spare" DASDs that are an integral part of the array. The spare DASD(s) is (are) unused during normal operations. On a DASD failure, the data that used to be on the failed DASD is rebuilt to a spare DASD. The array is said to leave degraded mode operation and re-enter normal mode operation (sometimes called "fault tolerant mode") as soon as the data on the failed DASD has been rebuilt to a spare DASD.
Traditional Sparing Referring now to figure 1, there is shown an array in which parity groups are coextensive with storage bounds. The array uses dedicated parity and spare DASDs in aid of reconstruction of missing data or parity onto the spare DASD according to the prior art.
The array in figure 1 comprises N+2 DASDs (for N=4).
One of the DASDs is a spare DASD that is unused in normal operation and the remaining 5 DASDs operate as a 4+P
array. This is termed "dedicated sparing". Each of the CA 02063897 1997-0~-28 '-90-079 10 2 0 6 3 8 9 ~
remaining N+l (5) DASDs is divided into some number K of blocks or block locations. This is called a "parity group" which consists of N data and one parity block, i.e. one block from each of N+1 DASDs. The array then can store K parity groups, each with N+l blocks or block locations.
In this invention and the prior art such as Dunphy et al , all logically related N+l blocks of data and parity are one to one mapped into N+l blocks of addressible storage. Consequently, the distinction between data and storage oriented parity groups disappears.
If data in any block location is lost, it can be reconstructed from the remaining N block locations of that parity group. When a DASD fails, K block locations from K different parity groups are lost. Each lost block location can be rebuilt using the corresponding block locations from the surviving DASDs. The rebuilt data is written to the spare DASD.
Referring again to figure 1, a failed DASD is shown as being crossed out. The reconstruction of the data contents depicted as the XORing of the operative contents from the N other DASDs. In the Figure 1 array, a DASD
failure requires the array to read 6 block locations from each of four DASDs, and to write 6 blocks to the spare DASD which now replaces the failed DASD. That is, a total of 30 DASD I/Os are needed to complete the rebuild.
Traditional sparing suffers two disadvantages.
First, it does not utilize the spare DASDs in normal operation. Second, the non-use of the spare DASD raises a doubt as to its operability when needed as a substitute for a failed DASD.
The second drawback could be overcome if the array controller were to periodically read and write data to various tracks of the spare DASDs in order to verify their correct operation.
Diestributed Sparing Referring now to figure 2, there is shown distributed sparing and distributed parity in a array CA 02063897 1997-0~-28 ~-90-079 11 comprising N+2 DASDs (N=4) as before. Some number of block locations (labelled sl through s6) are left unused in each DASD such that the total spare space on all the DASDs is equal to the capacity of a single DASD. Thus, this method leaves the same amount of spare space as current methods that use a dedicated spare DASD, but the spare space is distributed among all the DASDs rather than concentrated in a single DASD. Each parity group is extended to contain N data block, a parity block and a spare block. The data and parity block are also called "information blocks" to distinguish them from spare blocks.
As is apparent from figure 2, no two blocks from a parity group are on the same DASD. Therefore, if a DASD
fails, at most one information block from any parity group is lost and this lost block can be rebuilt from the remaining information blocks of the parity group on the other DASDs. The lost block from a parity group is rebuilt to the spare block for that parity group which is on another DASD.
For example, if DASD 3 were to fail in figure 2, block dl would be rebuilt to DASD 6, block d2 to DASD 5, block p3 to DASD 4, block d5 to DASD 2 and block d6 to DASD 1. Note that all the information blocks d4 and p4 survive the failure of DASD 3 and do not need to be rebuilt. In all, four blocks had to be read from each of 5 DASDs and one block had to be written to each of five DASDs, for a total of 25 DASD I/Os to complete the recovery process. This is better than the 30 DASD I/Os that were needed to complete a rebuild in the traditional sparing approach.
It follows that no two information blocks from a parity group end up on the same DASD following the rebuild, making it possible to tolerate another DASD
failure at this point.
Distributed Sparing Performance Consequences N+2 DASDs are used in normal mode (when no DASD has failed) as opposed to N+1 DASDs in current methods.
Typical values for N are between 4 and 10. With N=4, the CA 02063897 l997-0~-28 ~-90-079 12 2 0 6 3 8 9 7 distributed sparing scheme uses 6 DASDs in parallel instead of 5 and potentially improves performance by 20%
in normal mode. With N=10, distributed sparing could improve performance by 9% in normal mode.
When a DASD fails, the array is said to operate in degraded mode. Distributed sparing has better performance in degraded mode than traditional sparing for two reasons. First, more parallelism (N+l DASDs used instead of N in current methods) is involved. Secondly, in distributed sparing, only K-(K/(N+2)) blocks are lost (as opposed to K blocks for current methods) when a DASD
fails. In the earlier example, 5 blocks were lost when a DASD failed, whereas the traditional sparing approach lost 6 blocks when a DASD failed. Since accesses to lost blocks require extra accesses, the fewer blocks lost the better the overall performance.
Finally, distributed sparing has better performance during rebuild of lost data. In traditional sparing, the lost data is recovered to a single DASD which can be a bottleneck. With distributed sparing, the data is recovered in parallel to multiple DASDs so that no single DASD is a bottleneck. Furthermore, since less data is lost in this method, less data needs to be recovered.
This explains why, in the example, distributed sparing only needed 25 I/Os instead of the 30 I/Os required in traditional sparing.
Distributed sparing requires (N+l)*(K - (K/(N+2))) I/Os versus (N+l)*K I/Os for traditional sparing. The number of I/Os needed for rebuild has been reduced by the fraction (N+l)/(N+2).
As with traditional sparing, the rebuild of missing data and parity blocks preferably begins at cylinder zero of all DASDs and sweeps to the last cylinder. At the start of the sweep, the block lost from the first parity group would be rebuilt to DASD N+2; so DASD N+2 would be writing and the other DASDs would be reading. Then, for the second parity group, DASD N+l would be writing and the other DASDs would be reading, and so on to the last parity group. Thus, in figure 2, DASDs 1, 2, 4 and 5 would read block 1, while DASD 6 would write block 1.
CA 02063897 1997-0~-28 Then, DASDs 1, 2, 4 and 6 would read block 2 while DASD 5 would write block 2; and so on.
Spare Distribution, Rebuild, and Arm Stealing If distributed spare space were placed on the last few cylinders of each DASD, arm stealing would occur between the reads and writes (operations) to a DASD
during rebuild. This stems from the fact that the read operations require the arms to sweep from the first cylinder on down, whereas the write operations require the arms to be at the last cylinders.
Preferably, spare space should be distributed at several different physical locations on different DASDs in order to minimize arm stealing during rebuild.
However, physical address placement is involved in a trade off with other access performance factors. For instance, if all the spare space is at the extremities, then there would be less arm motion in normal operation of each drive.
Distrubuted Sparing Using Capacity of Multiple DASDs Referring now to figure 3, there is shown an array in which the spare capacity of up to two DASDs is distributed in a uniform pattern across N+3 DASDs. That is, in an array of N+3 DASDs each having a capacity of K
blocks/DASD, then 2*K blocks are distributed thereacross such that no more than two spare blocks are located on the same stripe or on the same DASD. Also, figure 3 depicts distribution of K*P parity blocks such that no more than a single parity block is located on the same stripe and on the same DASD.
Referring again to figure 3, it is apparent that an N+2 DASD array of fixed size formed where N=4 requires that a 4+P sized parity group tolerant of single DASD
failures would have to be reformatted and striped on a 3+P block basis. This would provide the 2*K spare capacity which could then be uniformly distributed.
It is considered well within the scope of this invention to extend the precept to higher numbers of failure tolerances.
CA 02063897 l997-0~-28 Distributed Sparing on Systems With Multiple Arrays Referring now to figures 4-8, there are shown distributed sparing among two or more failure independent arrays of DASDs. Multiple array configurations are significant where storage subsystem capacity is expanded incrementally. That is it is well appreciated within the scope of this invention to distribute and use spare capacity among failure independent addressible DASD
arrays so as to minimize both performance and cost impacts.
Storage Subsystem Expansion Where Each Array Inclused One DASD Distributed Spare Capacity Referring now to figure 4, there are depicted two 4+P arrays each having one DASD spare capacity distributed uniformly by or within the individual array.
If a DASD in an array were to fail, that DASD would be rebuild to the spare space distributed in that same array. That is, the spare space distributed in each array is available in common. Reference may be made to Dunphy et al, where dedicated spare DASDs were reserved in common among parity groups. However, Dunphy required that a spare DASD be dynamically and automatically switched to replace the failed DASD. Such switching is nowhere required in this invention.
Referring again to figure 4, if DASD 1 from array 1 were to fail, followed by DASD 3 from array 2, DASD 1 would be rebuilt to spare space in array 1 and DASD 3 would be rebuilt to spare space on array 2. However, if DASD 1 from array 1 fails first; this causes it to be rebuilt to spare space in array 1. Next, if DASD 4, also from array 1 fails, then no spare space is available on array 1. In this invention, the contents of the failed DASD 4 that is the second failure should be rebuilt onto the spare space on array 2.
Referring now to figure 5, there is depicted the failure of DASD 1. DASD 1 originally stored one data block from each of four parity groups (dl-d4), a parity block (p5) from a fifth parity group, and a spare or unassigned block (s6). The spares (sl-s6) are assigned CA 02063897 l997-0~-28 such that no synchronous address (the same address position across each of the DASDs in the array) has more than one spare block. The same applies to each DASD. In this aspect of the invention, DASD 1 is rebuilt such that blocks dl-d4 & p5 are determined by logically combining (XORing) the counterpart N other blocks of the parity group into a single block and writing the single block into a spare s(i) having the same synchronous address as the other blocks. Consequently, block d2 is formed by XORing the d2 blocks in the same address on DASDs 2, 3, and 6 and parity p2 on DASD 4. The rebuilt block d2 is then written in the spare position on DASD 5. This is repeated for all but the last block s6. Since s6 is a spare block, no logical combining and writing actions are required.
Referring now to figure 6, there is shown a second DASD failure (DASD 4) occurring in the same array (array 1) as the first failure. The sparing in both arrays is distributed such that each DASD has K/(N+2) spare blocks and and only one spare block can appear in the same synchronous address and DASD. This means that DASD 4 from array 1 can be rebuilt by logical combining the remaining N DASDs in the group into a single block and writing the block into counterpart spare blocks in array 2.
A storage subsystem can be expanded by adding one array at a time where each array comes with its own distributed spare space. A limitation of this expansion as configured is that it is not possible to have a single spare that is shared by multiple arrays. Therefore, the cost of sparing may be higher than that which can be afforded by a system or that which is appropriate for the system.
Referring again to figure 6, another limitation arises after the second failure, the blocks of array 1 are now scattered across 10 DASDs (4 surviving DASDs from array 1 and 6 from array 2) instead of the original 6 DASDs. Therefore, the simultaneous failure of any 2 DASDs from this group of 10 would cause data loss. In other words, as DASDs fail and are rebuilt to other arrays, the Mean-Time-To-Data-Loss (MTTDL) of the system gets CA 02063897 l997-0~-28 9-90-079 16 2 0 6 ~ 8 9 7 somewhat worse. The MTTDL is calculated as the probability that a second DASD fails shortly after the first one has failed and before it has been rebuilt.
Storage Subsystem Expansion Where Arrays Share DASD
Distributed spare Capacity Referring now to figure 7, array 1 is a 4+P with distributed sparing according to the precept of the invention. However, array 2 is also 4+P but without sparing.
If a DASD in array 1 were to fail, five blocks would be lost (one of the blocks is a spare block), and there would be five spare blocks on the other five DASDs in the array to rebuild the five blocks lost. If a DASD in array 2 were to fail, 6 blocks would be lost, and there would be 6 spare blocks on the 6 DASDs in array 1 to rebuild the 6 lost blocks. In this way, the distributed spare blocks in array 1 would be available to rebuild blocks of a single failure occurring in either array.
Advantages and Limitages of Partially Distributed Sparing Besides the advantage that such an approach allows us to share a single spare amongst multiple arrays and does not require each array to have a spare, it has the additional advantage that we can adjust the spare space requirements with system growth by allowing the choice of either adding an array with a spare or an array without a spare.
This aspect of the invention suffers the limitation that as DASDs fail and are replaced to other arrays, the MTTDL of the system will drop, until the broken DASDs are replaced and the dispersed data copied back.
Spare DASD Distributed Across Multiple Arrays Referring now to Table 2, the spare blocks of a single spare DASD are shared across multiple arrays. In this regard, the spare is shared across two 2+P arrays.
In the following embodiment, the distribution step and means are shown as they pertain to the laying out of CA 02063897 l997-0~-28 parity groups and single DASD s work of spare space across two arrays.
1 2 3 4 5 6 7 <-DASD
Dl Dl P1 dl dl pl Sl D2 P2 D2 d2 p2 S2 d2 P3 D3 D3 p3 S3 d3 d3 D4 D4 P4 S4 d4 d4 p4 D5 P5 S5 D5 d5 p5 d5 P6 S6 D6 D6 p6 d6 d6 S7 D7 D7 P7 d7 d7 p7 Table 2.
Data and parity blocks of array 1 are indicated by uppercase Ds and Ps; data and parity blocks of array 2 are indicated by lowercase ds and ps. Note that all the data and parity from array 1 are on DASDs 1, 2, 3 and 4 and that all the data and parity from array 2 are on DASDs 4, 5, 6 and 7. Therefore, the MTTDL of either array is the same as any 2+P array with distributed sparing.
The distribution step operates as follows:
Referring to Tables 2 and 3, Sl is placed on block 1 of DASD 7, S2 on block 2 of DASD 6, and so on. That is, the spares are rotated across the 7 DASDs shown in a uniform way. Next, the blocks of array 1 are stored on DASDs 1, 2 and 3 as:
1 2 3 <- DASD
D1 Dl P1 .etc...
CA 02063897 l997-0~-28 '-90-079 18 20$3897 Table 3.
However, if one of these blocks must be a spare as determined by the spare rotation above, then give priority to the spare and shift any data and parity blocks to the right to accommodate the spare. Thus as expressed in Table 4, the first four rows of the placement of array 1 are unaffected by spares, but the other three rows are affected by spare placement. The result causes array l s placement to become:
1 2 3 4 <- DASD
.etc...
Table 4.
Similarly, the data and parity blocks of array 2 are placed on DASDs 4, 5 and 6 except when they need to be shifted to the right to satisfy the placement of a spare or array 1.
The result has each DASD with 4 data blocks, 2 parity blocks and a spare block, so there is uniform distribution of spares and parity across all DASDs in the array.
Groups of Arrays In the above embodiment, the storage subsystem would expand by adding two 2+P arrays at a time. Relatedly, each two array unit would have the one spare DASDs worth of spare blocks to share between them.
CA 02063897 1997-0~-28 ''-90-079 19 2063897 In this embodiment, assume that the storage subsystem has expanded to 14 DASDs (2 array groups of 7 DASDs). Each array group of 7 DASDs has two 2+P arrays and 1 DASD equivalent of distributed spare blocks. Let the arrays in group 1 be array 1 and array 2; let the arrays in group 2 be array 3 and array 4. Consider that a DASD in array 1 fails; then it would be rebuilt to spare space in group 1.
Assume that a DASD in array 2 fails. Even though array 2 is part of group 1, we would allow it to be rebuilt to the spare space in group 2, since no spare space remains in group 1. This dispersal of data from a DASD in a first group to DASDs in other groups results in a decrease in MTTDL of the storage subsystem until the failed DASDs have been replaced and data has been copied back.
Storage System Expansion by Array Multiples Expansion of a storage subsystem by an array multiples bears resemblance to the previous distribution of groups and spare blocks. For example, suppose two 2+P
arrays share a single spare. The subsystem would be initially configured to support a single 2+P+S array. The system would expand by another 2+P array, resulting in a system with two 2+P arrays and a spare distributed amongst them. The layout of data and spares when there is only a single 2+P system is shown in Table 5.
1 2 3 4 <- DASD
Table 5.
~-90-079 20 2 0 6 3 8 9 7 As the next 2+P array is added to the same group, Table 6 depicts a distribution which shares the spare between the two 2+P arrays in the group without any data movement:
Dl D1 P1 dl dl pl S1 D2 P2 D2 d2 p2 S2 d2 P3 D3 D3 p3 S3 d3 d3 D4 D4 P4 S4 d4 d4 p4 D5 P5 S5 D5 d5 p5 d5 P6 S6 D6 D6 p6 d6 d6 S7 D7 D7 P7 d7 d7 p7 Table 6.
Advantageously, this distribution allows the K block capacity of a single spare DASD to be shared between multiple arrays, but does not require that expansion be in terms of multiple arrays. This avoids oversparing with respect to an ultimate sparing fraction objective.
Extensions The invention has been described where both the parity blocks and spare DASD capacity have been distributed. One extension is to preserve the parity blocks on a dedicated DASD and distribute just the spare capaclty .
Combinations of the aformentioned parity group and sparing distributions could be employed. It is possible to mix the distribution pattern in a multi-group array.
That is, the storage subsystem is capable of expansion by adding an array with spare, an array without a spare or multiple arrays with a shared spare between them at different times. The particular patterns used would be a function of the size of the system, the spare replacement policy, and the sparing ratio that is considered acceptable.
CA 02063897 l997-05-28 ~ 90_079 21 2063897 These and other extensions of the invention may be made without departing from the spirit and scope thereof as recited in the appended claims.
Field of the Invention This invention relates to DASD (Direct Access Storage Device) storage subsystems, and more particularly, to methods and means for managing spare DASD array capacity so as to optimize array operations in fault tolerant, degraded, and data rebuild modes.
Description of Related Art Arrays, Effect of Redundancy, Reading and Writing In the prior art, it is lcnown to read and write data + parity (as defined over the data) from and to a synchronous array of N data + P parity DASDs. The DASD
array arrangement increases the data rate by N * rate of a single DASD and increases logical track size by N * single DASD track length. Reference can be made to Patterson et al, "A Case For Redundant Arrays Of Inexpensive Dislcs (RAID)", Report No. UCB/CSD 87/391, December 1987, Computer Science Division, U. of California, Berlceley.
It is also lcnown that writing a data string to a DASD array includes segmenting data into N bloclcs (terrned striping), determining parity over the bloclcs, and storing the N data + P parity bloclcs at a synchronous address on counterpart failure independent DASDs. Likewise, reading a data string from an array involves copying N+P addressed bloclcs from a synchronous address on counterpart DASDs into a buffer, concatenating them, checldng parity, and serially transporting the concatenated blocks (string) from the buffer to the accessing CPU.
DASD Failure and MTBF
It is well recognized that aggregating DASDs into arrays does not improve the mean time between DASD failure. Rather, it decreases it. However, the combined CA 02063897 1997-0~-28 use of redundant information (parity), dynamic substitution of formatted spare DASDs for failed ones, and reconstruction of missing data onto the substituted spare, substantially increases the mean time between data unavailability by at least orders of magnitude. This is described in Park et al, "Providing Fault Tolerance In Parallel Secondary Storage Systems", Princeton University Report CS-TR-057-86, November 1986 and Dunphy et al, US
Pat. 4,914,656, "Disk Drive Memory", issued April 3, 1990 .
Two Usages of Parity Groups The term "parity group" has acquired data oriented and ~torage oriented usages.
In the data usage, "parity group" signifies a predetermined number of logically associated data blocks + a parity or equivalent redundant information defined over or derived from the data blocks. In the storage usage, "parity group" signifies a predetermined number of logically associated physical storage locations whose data contents determine the value of the redundant information.
Parity Spreading and DASD Arrays One method for managing DASD array accesses when operating in degraded mode, uses parity group as a logical association of data blocks as applied to Patterson s RAID 5 type DASD array. The data and storage boundaries of the parity group are not necessarily coextensive. The only limitation is that no two segmented blocks from the same data parity group be written onto the same DASD.
In contrast, Dunphy et al, US Pat. 4,914,656, "Disk Drive Memory", issued April 3, 1990, defines parity over data as in the above method but the storage boundaries are the same as that of the data parity group. They are maintained even in the presence of failure since a spare DASD substitutes for the failed DASD and missing data is rebuilt and rewritten onto the substituted spare.
CA 02063897 1997-0~-28 9_90_079 3 2063897 Clark et al, US Pat. 4,761,785, "Parity Spreading Enhanced Storage Açcess", issued August 2, 1988 discloses a non-synchronous DASD storage subsystem in which parity groups are defined over a range of storage and where data boundaries are NOT necessarily coextensive with storage boundaries.
In Clark et al, each data string is written into consecutive locations of a "storage" parity group. If the data string size exceeds the capacity of the group, then the residue of the data string is written into another "storage" parity group. If the data string is smaller than the group, then the remaining space may well be occupied by other data strings. Parity is taken across information occupying logically associated extents (range of address locations) of the DASDs forming the group.
Distributed Parity and Virtual or Distributed Sparing Clark et al also taught that the location of the parity blocks for counterpart storage parity groups could be distributed across DASDs in the subsystem with the only limitation that not all of the blocks be written on a single DASD. This was contrary to Ouchi, US Pat.
4,092,732, "System for Recovering Data Stored In A Failed Memory Unit", issued May 30, 1978, and Dunphy et al where parity is written to dedicated DASDs.
Another procedure for recovering parity protected data teaches the use of a virtual spare among non-synchronous DASDs where parity groups are defined across storage, and data and storage boundaries are not necessarily coextensive.
In this last-mentioned procedure, the CPU can read and write to a logical spare DASD. The logical addresses are then distributed in non-specific manner across one or more real DASDs. More particularly, it teaches that the CA 02063897 1997-0~-28 locations of parity blocks distributed as in Clark may be overwritten as if they were spare locations in the reconstruction of data being accessed after a DASD has failed. Usually, this means the parity block of the parity group covering the lost data.
In such a type of distributed sparing via writing reconstructed data into the group parity location, once recovery is complete, the system operates without parity.
This is inimical to a highly available, fault-tolerant system.
However, what this last-mentioned procedure failed to recognize was that:
a. two or more data blocks from the same storage parity group are written on the same DASD. If a DASD were to fail now, the DASD array subsystem would lose data, which is unacceptable.
b. data blocks of the same group formerly located on different DASDs now being resident upon the same DASD
cannot be read in parallel.
c. lowered throughput because of resource sharing among competitive processes occasioned by DASD arm stealing between the reading of the surviving data blocks to compute any missing data or parity block and the writing of a reconstructed data or parity block.
Summary of the Invention It is an object of this invention to devise a method and means for facilitating reconstruction of missing data and parity blocks and copying them back into spare block DASD locations.
It is a related object to devise a method and means in which the data and parity blocks forming parity groups together with spare blocks are distributed over array block locations according to at least one combinatorial design, each group having N data and P parity blocks.
Additionally, it is yet another object that said combinatorial designs minimize the number of accesses to reconstruct missing data and parity blocks and their copyback into spare block locations.
CA 02063897 1997-0~-28 It yet another object that the throughput be maximized during reconstruction and copyback of missing data or parity blocks and subsequent reference thereto.
Further, said method and means should be readily extensible to storage subsystems addressing two or more failure independent DASD arrays. Also, said method and means should be extensible such that the combinatorial design should be distributable over multiple failure independent arrays and with regard to different sparing ratios or fractions.
The foregoing objects are satisfied by a method and means for rebuilding portions of parity groups resident on a failed DASD, in a storage subsystem having a plurality of DASDS and where each DASD stores K blocks.
Also, each parity group includes N data, P parity, and S
spare blocks.
The method and means include configuring an array of N+P+S DASDs. Next, K parity groups are distributed in synchronous array addresses across subsets of the N+P+S
DASDs such that no two blocks from the same parity group reside on the same DASD. K*S blocks constituting spare storage are distributed across the array such that each DASD includes K*S/(N+P+S) spare blocks thereon.
Furthermore, no more than S spare storage blocks nor more than P parity blocks are stored on the same synchronous array address or on the same DASD.
In the event of a single DASD failure, for each of the (K-(K*S/(N+P+S))) affected parity groups, N+P+S-2 blocks belonging to the group are logically combined (XOR ed) from the other N+P+S-2 other DASDs into a single block. Each single block is written into a counterpart one of the remaining K*S*(N+P+S-l)/(N+P+S) spare blocks such that no two blocks of the same parity group are distributed on the same DASD.
In a first embodiment (where P=l, S=l), the method and means include configuring an array of N+2 DASDs. The next step involves distributing K-K/(N+2) parity groups in synchronous array addresses across subsets of N+1 DASDs of the array such that no two blocks from the same parity group reside on the same DASD. Concurrently, K
CA 02063897 1997-0~-28 '9-90-079 6 2 0 6 3 8 9 7 blocks as spare storage are distributed across the array such that each DASD includes K/(N+2) spare blocks thereon. As in the general case, no more than one spare storage block nor more than one parity block are stored on the same synchronous array address or on the same DASD.
In the event of a single DASD failure, for each K-K/(N+2) parity groups, N blocks belonging to the group are logically combined from N other DASDs into a single block. Each single block is written into a counterpart one of the remaining K*(N+1)/(N+2) spare blocks such that no two blocks of the same parity group are distributed on the same DASD.
In another embodiment (where P=l, S=2), 2*K blocks of spare storage and (K-2*K/(N+3)) parity blocks are distributed such that no more than two spare storage blocks nor more than one parity block are stored on the same synchronous array address nor on the same DASD. This permits rebuilding and writing missing data to a first series of spare blocks after a first DASD has failed and repeating the process in the rarer event that a second DASD should fail.
The method and apparatus of this invention are extensible to storage subsystems having multiple failure independent DASD arrays. Where multiple DASD failures occur in the same array, the missing data is first rebuilt on the spare space of the first array and the remaining missing data rebuilt on the spare space of the second array.
Where storage subsystem expansion involves one array with distributed spare capacity and an array without a spare, such capacity may be conveniently shared. Also, the blocks representing the capacity of one or more spare DASDs can be distributed across multiple arrays so that no synchronous address or DASD has more than one such spare block increment. Next, the parity groups can be written across their respective arrays in a rotated or block offset manner. This would permit a uniform sharing.
Other distributions of parity groups and sparing are disclosed which permit storage subsystem DASD array CA 02063897 1997-0~-28 ~9-90-079 7 2063897 expansion while maintaining a sparing fraction or ratio objective.
Brief Description of the Drawing Figure 1 shows parity groups coextensive with storage bounds using dedicated parity and spare DASDs and reconstruction of missing data or parity onto the spare DASD according to the prior art.
Figure 2 depicts distributing K parity groups {(4+P)<(N+2)} and K spare spaces over an array of N+2 DASDs of capacity K blocks or spaces/DASD permitting recovery from a single DASD failure according to the invention, no two elements of the same group nor space being located on the same DASD.
Figure 3 also sets out a distribution of K parity groups {(3+P)~(N+2)} and 2K spare spaces over an array of N+2 DASDs of capacity K blocks/DASD permitting recovery of up to two DASD failures according to the invention, no two elements of the same group nor spare space being located on the same DASD.
Figure 4 illustrates two DASD arrays with distributed sparing.
Figures 5 and 6 show failure of one or more DASDs in a first one of two arrays and the rebuilding of missing data on the distributed spare spaces across both arrays.
Figure 7 depicts partially distributed sparing.
Figure 8 depicts a synchronous array of N+2 DASDs attached to a CPU by way of an array control unit.
Despcription of the Preferred Embodiment DASD Array Architecture Referring now to figure 8, there is shown CPU 1 accessing DASDs 1 through N+2 over a path including channel 3, array controller 5, and cache 13. Controller 5 operatively secures synchronism and accesses among any N+1 at a time of the N+2 DASDs i.e. DASD 1 through DASD
N+2 over control path 7.
N+1 streams of data defining a predetermined number of consecutive bytes can be exchanged in parallel to CA 02063897 1997-0~-28 ~9-90-079 8 cache 13 over data path 15. The N+l streams of data may all be responsive to a single access (synchronous operation). Also, this invention is operative where each of the N+l streams may be responsive different accesses (asynchronous operations). Likewise, data can be exchanged serially by byte between CPU 1 and controller 5 over path 3 after a parallel to serial conversion in controller 5 in the read direction and a serial to parallel conversion in the write direction.
In the read direction, data is supplied from cache 13 to controller 5 via data paths 9 and 11. In the write direction, data is moved from the controller 5 to the cache 3 over paths 9 and 11.
Parity, DASD Failure and Sparing, and Robustness DASD arrays use parity to protect against single DASD failures. If a DASD fails, data that used to be on that DASD can be reconstructed, as needed, using the data and parity on the surviving DASDs. This is illustrated in Table 1 in a DASD array of five DASDs. In this diagram, Pi is a parity block that protects the four data blocks labelled Di. Such a DASD array is called a 4+P array, since there is one parity block for every four data blocks.
DASD 3 Dl D2 P3 D4 D5 Table 1.
Only one track (consisting of five blocks) is shown from each of the DASDs. Pl contains the parity or exclusive OR of the blocks labeled Dl on all the data DASDs. Similarly, P2 is the exclusive OR of the blocks labeled D2 on all the DASDs, and so on. It should be noted that the parity blocks are distributed amongst all the drives to balance the workload.
Such a DASD array is robust against single DASD
failures. If DASD 1 were to fail, data on it can be recreated by reading data and parity from the remaining four DASDs and performing the appropriate exclusive OR
operations.
An array is said to enter "degraded mode" when a DASD in the array fails, because the performance and reliability of the array becomes degraded. Performance is degraded since every DASD access to a block on the failed DASD now requires DASD accesses to other DASDs in the array for reconstructing the block that is no longer accessible. Reliability is degraded, since if a second DASD fails before the failed DASD is replaced and the data on it reconstructed, the array will lose data.
In the specification of this invention, the term "reliability" indicates the degree of immunity from data loss it possesses. The higher the immunity from data loss, or the higher the mean time to data loss (MTTDL), the higher the reliability of the array.
To minimize the probability of losing data and the length of time the array operates with degraded performance, arrays sometimes use "hot spare" DASDs that are an integral part of the array. The spare DASD(s) is (are) unused during normal operations. On a DASD failure, the data that used to be on the failed DASD is rebuilt to a spare DASD. The array is said to leave degraded mode operation and re-enter normal mode operation (sometimes called "fault tolerant mode") as soon as the data on the failed DASD has been rebuilt to a spare DASD.
Traditional Sparing Referring now to figure 1, there is shown an array in which parity groups are coextensive with storage bounds. The array uses dedicated parity and spare DASDs in aid of reconstruction of missing data or parity onto the spare DASD according to the prior art.
The array in figure 1 comprises N+2 DASDs (for N=4).
One of the DASDs is a spare DASD that is unused in normal operation and the remaining 5 DASDs operate as a 4+P
array. This is termed "dedicated sparing". Each of the CA 02063897 1997-0~-28 '-90-079 10 2 0 6 3 8 9 ~
remaining N+l (5) DASDs is divided into some number K of blocks or block locations. This is called a "parity group" which consists of N data and one parity block, i.e. one block from each of N+1 DASDs. The array then can store K parity groups, each with N+l blocks or block locations.
In this invention and the prior art such as Dunphy et al , all logically related N+l blocks of data and parity are one to one mapped into N+l blocks of addressible storage. Consequently, the distinction between data and storage oriented parity groups disappears.
If data in any block location is lost, it can be reconstructed from the remaining N block locations of that parity group. When a DASD fails, K block locations from K different parity groups are lost. Each lost block location can be rebuilt using the corresponding block locations from the surviving DASDs. The rebuilt data is written to the spare DASD.
Referring again to figure 1, a failed DASD is shown as being crossed out. The reconstruction of the data contents depicted as the XORing of the operative contents from the N other DASDs. In the Figure 1 array, a DASD
failure requires the array to read 6 block locations from each of four DASDs, and to write 6 blocks to the spare DASD which now replaces the failed DASD. That is, a total of 30 DASD I/Os are needed to complete the rebuild.
Traditional sparing suffers two disadvantages.
First, it does not utilize the spare DASDs in normal operation. Second, the non-use of the spare DASD raises a doubt as to its operability when needed as a substitute for a failed DASD.
The second drawback could be overcome if the array controller were to periodically read and write data to various tracks of the spare DASDs in order to verify their correct operation.
Diestributed Sparing Referring now to figure 2, there is shown distributed sparing and distributed parity in a array CA 02063897 1997-0~-28 ~-90-079 11 comprising N+2 DASDs (N=4) as before. Some number of block locations (labelled sl through s6) are left unused in each DASD such that the total spare space on all the DASDs is equal to the capacity of a single DASD. Thus, this method leaves the same amount of spare space as current methods that use a dedicated spare DASD, but the spare space is distributed among all the DASDs rather than concentrated in a single DASD. Each parity group is extended to contain N data block, a parity block and a spare block. The data and parity block are also called "information blocks" to distinguish them from spare blocks.
As is apparent from figure 2, no two blocks from a parity group are on the same DASD. Therefore, if a DASD
fails, at most one information block from any parity group is lost and this lost block can be rebuilt from the remaining information blocks of the parity group on the other DASDs. The lost block from a parity group is rebuilt to the spare block for that parity group which is on another DASD.
For example, if DASD 3 were to fail in figure 2, block dl would be rebuilt to DASD 6, block d2 to DASD 5, block p3 to DASD 4, block d5 to DASD 2 and block d6 to DASD 1. Note that all the information blocks d4 and p4 survive the failure of DASD 3 and do not need to be rebuilt. In all, four blocks had to be read from each of 5 DASDs and one block had to be written to each of five DASDs, for a total of 25 DASD I/Os to complete the recovery process. This is better than the 30 DASD I/Os that were needed to complete a rebuild in the traditional sparing approach.
It follows that no two information blocks from a parity group end up on the same DASD following the rebuild, making it possible to tolerate another DASD
failure at this point.
Distributed Sparing Performance Consequences N+2 DASDs are used in normal mode (when no DASD has failed) as opposed to N+1 DASDs in current methods.
Typical values for N are between 4 and 10. With N=4, the CA 02063897 l997-0~-28 ~-90-079 12 2 0 6 3 8 9 7 distributed sparing scheme uses 6 DASDs in parallel instead of 5 and potentially improves performance by 20%
in normal mode. With N=10, distributed sparing could improve performance by 9% in normal mode.
When a DASD fails, the array is said to operate in degraded mode. Distributed sparing has better performance in degraded mode than traditional sparing for two reasons. First, more parallelism (N+l DASDs used instead of N in current methods) is involved. Secondly, in distributed sparing, only K-(K/(N+2)) blocks are lost (as opposed to K blocks for current methods) when a DASD
fails. In the earlier example, 5 blocks were lost when a DASD failed, whereas the traditional sparing approach lost 6 blocks when a DASD failed. Since accesses to lost blocks require extra accesses, the fewer blocks lost the better the overall performance.
Finally, distributed sparing has better performance during rebuild of lost data. In traditional sparing, the lost data is recovered to a single DASD which can be a bottleneck. With distributed sparing, the data is recovered in parallel to multiple DASDs so that no single DASD is a bottleneck. Furthermore, since less data is lost in this method, less data needs to be recovered.
This explains why, in the example, distributed sparing only needed 25 I/Os instead of the 30 I/Os required in traditional sparing.
Distributed sparing requires (N+l)*(K - (K/(N+2))) I/Os versus (N+l)*K I/Os for traditional sparing. The number of I/Os needed for rebuild has been reduced by the fraction (N+l)/(N+2).
As with traditional sparing, the rebuild of missing data and parity blocks preferably begins at cylinder zero of all DASDs and sweeps to the last cylinder. At the start of the sweep, the block lost from the first parity group would be rebuilt to DASD N+2; so DASD N+2 would be writing and the other DASDs would be reading. Then, for the second parity group, DASD N+l would be writing and the other DASDs would be reading, and so on to the last parity group. Thus, in figure 2, DASDs 1, 2, 4 and 5 would read block 1, while DASD 6 would write block 1.
CA 02063897 1997-0~-28 Then, DASDs 1, 2, 4 and 6 would read block 2 while DASD 5 would write block 2; and so on.
Spare Distribution, Rebuild, and Arm Stealing If distributed spare space were placed on the last few cylinders of each DASD, arm stealing would occur between the reads and writes (operations) to a DASD
during rebuild. This stems from the fact that the read operations require the arms to sweep from the first cylinder on down, whereas the write operations require the arms to be at the last cylinders.
Preferably, spare space should be distributed at several different physical locations on different DASDs in order to minimize arm stealing during rebuild.
However, physical address placement is involved in a trade off with other access performance factors. For instance, if all the spare space is at the extremities, then there would be less arm motion in normal operation of each drive.
Distrubuted Sparing Using Capacity of Multiple DASDs Referring now to figure 3, there is shown an array in which the spare capacity of up to two DASDs is distributed in a uniform pattern across N+3 DASDs. That is, in an array of N+3 DASDs each having a capacity of K
blocks/DASD, then 2*K blocks are distributed thereacross such that no more than two spare blocks are located on the same stripe or on the same DASD. Also, figure 3 depicts distribution of K*P parity blocks such that no more than a single parity block is located on the same stripe and on the same DASD.
Referring again to figure 3, it is apparent that an N+2 DASD array of fixed size formed where N=4 requires that a 4+P sized parity group tolerant of single DASD
failures would have to be reformatted and striped on a 3+P block basis. This would provide the 2*K spare capacity which could then be uniformly distributed.
It is considered well within the scope of this invention to extend the precept to higher numbers of failure tolerances.
CA 02063897 l997-0~-28 Distributed Sparing on Systems With Multiple Arrays Referring now to figures 4-8, there are shown distributed sparing among two or more failure independent arrays of DASDs. Multiple array configurations are significant where storage subsystem capacity is expanded incrementally. That is it is well appreciated within the scope of this invention to distribute and use spare capacity among failure independent addressible DASD
arrays so as to minimize both performance and cost impacts.
Storage Subsystem Expansion Where Each Array Inclused One DASD Distributed Spare Capacity Referring now to figure 4, there are depicted two 4+P arrays each having one DASD spare capacity distributed uniformly by or within the individual array.
If a DASD in an array were to fail, that DASD would be rebuild to the spare space distributed in that same array. That is, the spare space distributed in each array is available in common. Reference may be made to Dunphy et al, where dedicated spare DASDs were reserved in common among parity groups. However, Dunphy required that a spare DASD be dynamically and automatically switched to replace the failed DASD. Such switching is nowhere required in this invention.
Referring again to figure 4, if DASD 1 from array 1 were to fail, followed by DASD 3 from array 2, DASD 1 would be rebuilt to spare space in array 1 and DASD 3 would be rebuilt to spare space on array 2. However, if DASD 1 from array 1 fails first; this causes it to be rebuilt to spare space in array 1. Next, if DASD 4, also from array 1 fails, then no spare space is available on array 1. In this invention, the contents of the failed DASD 4 that is the second failure should be rebuilt onto the spare space on array 2.
Referring now to figure 5, there is depicted the failure of DASD 1. DASD 1 originally stored one data block from each of four parity groups (dl-d4), a parity block (p5) from a fifth parity group, and a spare or unassigned block (s6). The spares (sl-s6) are assigned CA 02063897 l997-0~-28 such that no synchronous address (the same address position across each of the DASDs in the array) has more than one spare block. The same applies to each DASD. In this aspect of the invention, DASD 1 is rebuilt such that blocks dl-d4 & p5 are determined by logically combining (XORing) the counterpart N other blocks of the parity group into a single block and writing the single block into a spare s(i) having the same synchronous address as the other blocks. Consequently, block d2 is formed by XORing the d2 blocks in the same address on DASDs 2, 3, and 6 and parity p2 on DASD 4. The rebuilt block d2 is then written in the spare position on DASD 5. This is repeated for all but the last block s6. Since s6 is a spare block, no logical combining and writing actions are required.
Referring now to figure 6, there is shown a second DASD failure (DASD 4) occurring in the same array (array 1) as the first failure. The sparing in both arrays is distributed such that each DASD has K/(N+2) spare blocks and and only one spare block can appear in the same synchronous address and DASD. This means that DASD 4 from array 1 can be rebuilt by logical combining the remaining N DASDs in the group into a single block and writing the block into counterpart spare blocks in array 2.
A storage subsystem can be expanded by adding one array at a time where each array comes with its own distributed spare space. A limitation of this expansion as configured is that it is not possible to have a single spare that is shared by multiple arrays. Therefore, the cost of sparing may be higher than that which can be afforded by a system or that which is appropriate for the system.
Referring again to figure 6, another limitation arises after the second failure, the blocks of array 1 are now scattered across 10 DASDs (4 surviving DASDs from array 1 and 6 from array 2) instead of the original 6 DASDs. Therefore, the simultaneous failure of any 2 DASDs from this group of 10 would cause data loss. In other words, as DASDs fail and are rebuilt to other arrays, the Mean-Time-To-Data-Loss (MTTDL) of the system gets CA 02063897 l997-0~-28 9-90-079 16 2 0 6 ~ 8 9 7 somewhat worse. The MTTDL is calculated as the probability that a second DASD fails shortly after the first one has failed and before it has been rebuilt.
Storage Subsystem Expansion Where Arrays Share DASD
Distributed spare Capacity Referring now to figure 7, array 1 is a 4+P with distributed sparing according to the precept of the invention. However, array 2 is also 4+P but without sparing.
If a DASD in array 1 were to fail, five blocks would be lost (one of the blocks is a spare block), and there would be five spare blocks on the other five DASDs in the array to rebuild the five blocks lost. If a DASD in array 2 were to fail, 6 blocks would be lost, and there would be 6 spare blocks on the 6 DASDs in array 1 to rebuild the 6 lost blocks. In this way, the distributed spare blocks in array 1 would be available to rebuild blocks of a single failure occurring in either array.
Advantages and Limitages of Partially Distributed Sparing Besides the advantage that such an approach allows us to share a single spare amongst multiple arrays and does not require each array to have a spare, it has the additional advantage that we can adjust the spare space requirements with system growth by allowing the choice of either adding an array with a spare or an array without a spare.
This aspect of the invention suffers the limitation that as DASDs fail and are replaced to other arrays, the MTTDL of the system will drop, until the broken DASDs are replaced and the dispersed data copied back.
Spare DASD Distributed Across Multiple Arrays Referring now to Table 2, the spare blocks of a single spare DASD are shared across multiple arrays. In this regard, the spare is shared across two 2+P arrays.
In the following embodiment, the distribution step and means are shown as they pertain to the laying out of CA 02063897 l997-0~-28 parity groups and single DASD s work of spare space across two arrays.
1 2 3 4 5 6 7 <-DASD
Dl Dl P1 dl dl pl Sl D2 P2 D2 d2 p2 S2 d2 P3 D3 D3 p3 S3 d3 d3 D4 D4 P4 S4 d4 d4 p4 D5 P5 S5 D5 d5 p5 d5 P6 S6 D6 D6 p6 d6 d6 S7 D7 D7 P7 d7 d7 p7 Table 2.
Data and parity blocks of array 1 are indicated by uppercase Ds and Ps; data and parity blocks of array 2 are indicated by lowercase ds and ps. Note that all the data and parity from array 1 are on DASDs 1, 2, 3 and 4 and that all the data and parity from array 2 are on DASDs 4, 5, 6 and 7. Therefore, the MTTDL of either array is the same as any 2+P array with distributed sparing.
The distribution step operates as follows:
Referring to Tables 2 and 3, Sl is placed on block 1 of DASD 7, S2 on block 2 of DASD 6, and so on. That is, the spares are rotated across the 7 DASDs shown in a uniform way. Next, the blocks of array 1 are stored on DASDs 1, 2 and 3 as:
1 2 3 <- DASD
D1 Dl P1 .etc...
CA 02063897 l997-0~-28 '-90-079 18 20$3897 Table 3.
However, if one of these blocks must be a spare as determined by the spare rotation above, then give priority to the spare and shift any data and parity blocks to the right to accommodate the spare. Thus as expressed in Table 4, the first four rows of the placement of array 1 are unaffected by spares, but the other three rows are affected by spare placement. The result causes array l s placement to become:
1 2 3 4 <- DASD
.etc...
Table 4.
Similarly, the data and parity blocks of array 2 are placed on DASDs 4, 5 and 6 except when they need to be shifted to the right to satisfy the placement of a spare or array 1.
The result has each DASD with 4 data blocks, 2 parity blocks and a spare block, so there is uniform distribution of spares and parity across all DASDs in the array.
Groups of Arrays In the above embodiment, the storage subsystem would expand by adding two 2+P arrays at a time. Relatedly, each two array unit would have the one spare DASDs worth of spare blocks to share between them.
CA 02063897 1997-0~-28 ''-90-079 19 2063897 In this embodiment, assume that the storage subsystem has expanded to 14 DASDs (2 array groups of 7 DASDs). Each array group of 7 DASDs has two 2+P arrays and 1 DASD equivalent of distributed spare blocks. Let the arrays in group 1 be array 1 and array 2; let the arrays in group 2 be array 3 and array 4. Consider that a DASD in array 1 fails; then it would be rebuilt to spare space in group 1.
Assume that a DASD in array 2 fails. Even though array 2 is part of group 1, we would allow it to be rebuilt to the spare space in group 2, since no spare space remains in group 1. This dispersal of data from a DASD in a first group to DASDs in other groups results in a decrease in MTTDL of the storage subsystem until the failed DASDs have been replaced and data has been copied back.
Storage System Expansion by Array Multiples Expansion of a storage subsystem by an array multiples bears resemblance to the previous distribution of groups and spare blocks. For example, suppose two 2+P
arrays share a single spare. The subsystem would be initially configured to support a single 2+P+S array. The system would expand by another 2+P array, resulting in a system with two 2+P arrays and a spare distributed amongst them. The layout of data and spares when there is only a single 2+P system is shown in Table 5.
1 2 3 4 <- DASD
Table 5.
~-90-079 20 2 0 6 3 8 9 7 As the next 2+P array is added to the same group, Table 6 depicts a distribution which shares the spare between the two 2+P arrays in the group without any data movement:
Dl D1 P1 dl dl pl S1 D2 P2 D2 d2 p2 S2 d2 P3 D3 D3 p3 S3 d3 d3 D4 D4 P4 S4 d4 d4 p4 D5 P5 S5 D5 d5 p5 d5 P6 S6 D6 D6 p6 d6 d6 S7 D7 D7 P7 d7 d7 p7 Table 6.
Advantageously, this distribution allows the K block capacity of a single spare DASD to be shared between multiple arrays, but does not require that expansion be in terms of multiple arrays. This avoids oversparing with respect to an ultimate sparing fraction objective.
Extensions The invention has been described where both the parity blocks and spare DASD capacity have been distributed. One extension is to preserve the parity blocks on a dedicated DASD and distribute just the spare capaclty .
Combinations of the aformentioned parity group and sparing distributions could be employed. It is possible to mix the distribution pattern in a multi-group array.
That is, the storage subsystem is capable of expansion by adding an array with spare, an array without a spare or multiple arrays with a shared spare between them at different times. The particular patterns used would be a function of the size of the system, the spare replacement policy, and the sparing ratio that is considered acceptable.
CA 02063897 l997-05-28 ~ 90_079 21 2063897 These and other extensions of the invention may be made without departing from the spirit and scope thereof as recited in the appended claims.
Claims (24)
1. In a storage subsystem having a plurality of DASDs, a method for rebuilding portions of parity groups resident on a failed one of said DASDs, each parity group including N data, P parity, and S spare blocks, each DASD
storing K blocks, comprising the steps of:
(a) configuring an array of N+P+S DASDs;
(b) distributing K parity groups in synchronous array addresses across subsets of N+P DASDs of the array such that no two blocks from the same parity group reside on the same DASD, each DASD storing data or parity blocks from (K-K*S/(N+P+S)) parity groups, and distributing K*S
blocks as spare storage across the array such that each DASD includes K*S/(N+P+S) spare blocks thereon; and (c) in the event of a single DASD failure, for each of the K-K*S/(N+P+S) parity groups on the failed DASD, logically combining N+P+S-2 blocks belonging to the group from N+P+S-2 other DASDs into a single block, and, writing said single block into a counterpart one of the remaining K*S*(N+P+S-1)/(N+P+S) spare blocks such that no two blocks of the same parity group are distributed on the same DASD.
storing K blocks, comprising the steps of:
(a) configuring an array of N+P+S DASDs;
(b) distributing K parity groups in synchronous array addresses across subsets of N+P DASDs of the array such that no two blocks from the same parity group reside on the same DASD, each DASD storing data or parity blocks from (K-K*S/(N+P+S)) parity groups, and distributing K*S
blocks as spare storage across the array such that each DASD includes K*S/(N+P+S) spare blocks thereon; and (c) in the event of a single DASD failure, for each of the K-K*S/(N+P+S) parity groups on the failed DASD, logically combining N+P+S-2 blocks belonging to the group from N+P+S-2 other DASDs into a single block, and, writing said single block into a counterpart one of the remaining K*S*(N+P+S-1)/(N+P+S) spare blocks such that no two blocks of the same parity group are distributed on the same DASD.
2. In a storage subsystem having a plurality of DASDs, a method for rebuilding portions of parity groups resident on a failed one of said DASDs, each parity group including N data and 1 parity blocks, each DASD storing K
blocks, comprising the steps of:
(a) configuring an array of N+2 DASDs;
(b) distributing K parity groups in synchronous array addresses across subsets of N+1 DASDs of the array such that no two blocks from the same parity group reside on the same DASD, each DASD storing K-K/(N+2) data or parity blocks, and distributing K blocks as spare storage across the array such that each DASD includes K/(N+2) spare blocks thereon; and (c) in the event of a single DASD failure, for each of the K-K/(N+2) parity groups, logically combining N
blocks belonging to the group from N other DASDs into a single block, and, writing said single block into a counterpart one of the remaining K*(N+1)/(N+2) spare blocks such that no two blocks of the same parity group are distributed on the same DASD.
blocks, comprising the steps of:
(a) configuring an array of N+2 DASDs;
(b) distributing K parity groups in synchronous array addresses across subsets of N+1 DASDs of the array such that no two blocks from the same parity group reside on the same DASD, each DASD storing K-K/(N+2) data or parity blocks, and distributing K blocks as spare storage across the array such that each DASD includes K/(N+2) spare blocks thereon; and (c) in the event of a single DASD failure, for each of the K-K/(N+2) parity groups, logically combining N
blocks belonging to the group from N other DASDs into a single block, and, writing said single block into a counterpart one of the remaining K*(N+1)/(N+2) spare blocks such that no two blocks of the same parity group are distributed on the same DASD.
3. The method according to claim 2, wherein each parity group is written into N+1 storage locations, and upon failure of a single DASD and rebuilding of said parity groups, only K-(K/(N+2)) storage locations are rendered unavailable.
4. The method according to claim 2, wherein all array DASDs other than failed DASD are addressable and responsive to access commands whether operated in fault tolerant or degraded modes.
5. The method according to claim 2, wherein the steps of distributing up to K parity groups and K spare storage blocks further comprises the steps of:
distributing said parity groups and spare blocks across N+2 DASDs such that no more than one spare storage block nor more than one parity block are stored on the same synchronous array address or on the same DASD.
distributing said parity groups and spare blocks across N+2 DASDs such that no more than one spare storage block nor more than one parity block are stored on the same synchronous array address or on the same DASD.
6. The method according to claim 2, wherein steps (b) and (c) further comprise the steps of:
distributing 2*K blocks of spare storage and K*P
parity blocks such that no more than two spare storage blocks nor more than P parity blocks are stored on the same synchronous array address or on the same DASD.
distributing 2*K blocks of spare storage and K*P
parity blocks such that no more than two spare storage blocks nor more than P parity blocks are stored on the same synchronous array address or on the same DASD.
7. The method according to claim 2, wherein the parity block from each of the K groups is written to a dedicated one of the N+2 DASDs.
8. The method according to claim 2, wherein each parity group is accessed concurrently from a selective subset of N+1 of the N+2 DASDs.
9. The method according to claim 2, wherein each parity group is accessed non-concurrently from a selective subset of N+1 of the N+2 DASDs.
10. The method according to claim 2, wherein each DASD
includes cyclic track storage means of m tracks; and means for moving from track to track and reading or writing data or parity blocks selectively along one or more tracks; and further wherein the method step (c) includes the steps of:
(1) positioning the moving means to a predetermined location on the cyclic track storage means of each of the remaining N+1 DASDs and traversing all m tracks starting from the predetermined location;
(2) at the start of the traverse, logically combining and writing the block lost from the first parity group onto the spare block of (N+2)nd DASD
concurrent with a reading operation performed by the remaining N other DASDs;
(3) continuing logically combining and writing the block lost from the second parity group on the spare block of the (N+1)st DASD concurrent with a reading operation performed by the remaining N DASDs; and (4) repeating step (3) until each block stored on the failed DASD from the K-K/(N+2) parity groups is recreated and rewritten into a counterpart spare block across each of the remaining DASDs.
includes cyclic track storage means of m tracks; and means for moving from track to track and reading or writing data or parity blocks selectively along one or more tracks; and further wherein the method step (c) includes the steps of:
(1) positioning the moving means to a predetermined location on the cyclic track storage means of each of the remaining N+1 DASDs and traversing all m tracks starting from the predetermined location;
(2) at the start of the traverse, logically combining and writing the block lost from the first parity group onto the spare block of (N+2)nd DASD
concurrent with a reading operation performed by the remaining N other DASDs;
(3) continuing logically combining and writing the block lost from the second parity group on the spare block of the (N+1)st DASD concurrent with a reading operation performed by the remaining N DASDs; and (4) repeating step (3) until each block stored on the failed DASD from the K-K/(N+2) parity groups is recreated and rewritten into a counterpart spare block across each of the remaining DASDs.
11. A storage subsystem for accessing parity groups of N
data blocks and 1 parity block each, comprising:
an array formed from N+2 DASDs, each DASD having the storing K blocks;
first means for distributing N+1 blocks of each of K
parity groups across counterpart subsets of N+1 DASDs selected from the array such that no two blocks from the same parity group is stored on the same DASD;
means for distributing K blocks of storage as spare blocks such that each array DASD reserves K/(N+2) blocks thereon;
second means for identifying any single DASD
failure; and means responsive to any single DASD failure identified by the seconds means for processing each of up to K-K/(N+2) parity groups by logically combining N
blocks belonging to the group from N other DASDs into a single block, and, writing said single block into a counterpart one of the remaining up to K*(N+1)/(N+2) spare blocks such that no two blocks of the same parity group are distributed on the same DASD.
data blocks and 1 parity block each, comprising:
an array formed from N+2 DASDs, each DASD having the storing K blocks;
first means for distributing N+1 blocks of each of K
parity groups across counterpart subsets of N+1 DASDs selected from the array such that no two blocks from the same parity group is stored on the same DASD;
means for distributing K blocks of storage as spare blocks such that each array DASD reserves K/(N+2) blocks thereon;
second means for identifying any single DASD
failure; and means responsive to any single DASD failure identified by the seconds means for processing each of up to K-K/(N+2) parity groups by logically combining N
blocks belonging to the group from N other DASDs into a single block, and, writing said single block into a counterpart one of the remaining up to K*(N+1)/(N+2) spare blocks such that no two blocks of the same parity group are distributed on the same DASD.
12. The storage subsystem according to claim 11, wherein the K spare blocks are distributed such that no two blocks occupy the same array address and the same DASD.
13. The storage subsystem according to claim 11, wherein upon failure of a single DASD and rebuilding of said parity groups, only K-(K/(N+2)) block storage locations are rendered unavailable for array use.
14. The storage subsystem according to claim 11, wherein the parity block from each of the K-K/(N+2) groups is written to a dedicated one of the N+2 DASDs.
15. The storage subsystem according to claim 11, wherein each DASD includes cyclic track storage means of m tracks; and means for moving from track to track and reading or writing data or parity blocks selectively along one or more tracks; and further wherein the means for logically combining and writing the K parity groups includes:
means for positioning the moving means to a predetermined location on the cyclic track storage means of each of the remaining N+1 DASDs and for traversing all m tracks starting from the predetermined location;
third means at the start of the traverse, for logically combining and for writing the block lost from the first parity group on the spare block of (N+2)nd DASD
concurrent with a reading operation performed by the remaining N other DASDs; and fourth means including the third means for continuing logically combining and writing the block lost from the second parity group on the spare block of the (N+1)st DASD concurrent with a reading operation performed by the remaining N DASDs, and for repeating the combining and writing until each block stored on the failed DASD from the K parity groups is recreated and rewritten into a counterpart spare block across the remaining DASDs.
means for positioning the moving means to a predetermined location on the cyclic track storage means of each of the remaining N+1 DASDs and for traversing all m tracks starting from the predetermined location;
third means at the start of the traverse, for logically combining and for writing the block lost from the first parity group on the spare block of (N+2)nd DASD
concurrent with a reading operation performed by the remaining N other DASDs; and fourth means including the third means for continuing logically combining and writing the block lost from the second parity group on the spare block of the (N+1)st DASD concurrent with a reading operation performed by the remaining N DASDs, and for repeating the combining and writing until each block stored on the failed DASD from the K parity groups is recreated and rewritten into a counterpart spare block across the remaining DASDs.
16. A storage subsystem comprising:
an array formed from at least N+P+2 DASDs, each DASD
having the capacity to store K blocks;
first means for distributing K parity groups across any N+P DASD subset of the N+P+2 DASDs in synchronous addresses, each parity group consisting of N data blocks + P parity blocks such that no two blocks from the same parity group are stored on the same DASD;
means for reserving and distributing the capacity equivalent of up to 2*K blocks of storage as spare blocks across the array of N+P+2 DASDs such that no more than two spare storage locations nor more than one parity block are stored on the same synchronous address or on the same DASD, second means for identifying up to any two DASD
failures in said array; and means responsive to identification by said second means of up to any two DASD failures for logically combining the K-2*K/(N+P+2) parity groups from N
remaining DASDs into single blocks, and writing the single blocks into the 2*(K-1) blocks of remaining reserved spare storage such that no two portions of the same parity group are distributed on the same DASD.
an array formed from at least N+P+2 DASDs, each DASD
having the capacity to store K blocks;
first means for distributing K parity groups across any N+P DASD subset of the N+P+2 DASDs in synchronous addresses, each parity group consisting of N data blocks + P parity blocks such that no two blocks from the same parity group are stored on the same DASD;
means for reserving and distributing the capacity equivalent of up to 2*K blocks of storage as spare blocks across the array of N+P+2 DASDs such that no more than two spare storage locations nor more than one parity block are stored on the same synchronous address or on the same DASD, second means for identifying up to any two DASD
failures in said array; and means responsive to identification by said second means of up to any two DASD failures for logically combining the K-2*K/(N+P+2) parity groups from N
remaining DASDs into single blocks, and writing the single blocks into the 2*(K-1) blocks of remaining reserved spare storage such that no two portions of the same parity group are distributed on the same DASD.
17. The storage subsystem according to claim 16, wherein said system further comprises means for distributing K*P
parity blocks such that not more than one parity block is located at the same synchronous address nor on the same DASD.
parity blocks such that not more than one parity block is located at the same synchronous address nor on the same DASD.
18. The storage subsystem according to claim 16, wherein each parity group is accessed concurrently from a selective N DASD subset of the N+2 DASDs.
19. A storage subsystem comprising:
a first and a second failure independent array each formed from at least N+P-1 DASDs, each DASD having the capacity to store K blocks;
first means for distributing K parity groups across N+P+1 DASDs of either the first or second arrays mutually exclusively, each parity group consisting of N data blocks + P parity blocks such that no two blocks from the same parity group are stored on the same DASD;
means for distributing K blocks of storage as spare blocks across N+P+1 DASDs of the first array and K blocks of storage as spare blocks across N+P+1 DASDs of the second array such that in each array only one storage block resides at each synchronous address and on each DASD;
second means for identifying a first or a second DASD failure occurring in either the first or second arrays;
means responsive to any single DASD failure identified by the second means for processing, in the array in which the failure occurred, each of the K-K/(N+P+1) parity groups by logically combining N+P-1 blocks belonging to the group from N+P-1 other DASDs into a single block, and, writing said single block into a counterpart one of the remaining K*(N+P)/(N+P+1) spare blocks such that no two blocks of the same parity group are distributed on the same DASD; and means responsive to any second DASD failure identified in the same or other array by the second means for processing, in the array having available spare blocks, each of the K-K/(N+2) parity groups by logically combining N+P-1 blocks belonging to the group from N+P-1 other DASDs in the array in which the second failure occurred into a single block, and, writing said single block into a counterpart one of the remaining K*(N+P)/(N+P+1) spare blocks such that no two blocks of the same parity group are distributed on the same DASD.
a first and a second failure independent array each formed from at least N+P-1 DASDs, each DASD having the capacity to store K blocks;
first means for distributing K parity groups across N+P+1 DASDs of either the first or second arrays mutually exclusively, each parity group consisting of N data blocks + P parity blocks such that no two blocks from the same parity group are stored on the same DASD;
means for distributing K blocks of storage as spare blocks across N+P+1 DASDs of the first array and K blocks of storage as spare blocks across N+P+1 DASDs of the second array such that in each array only one storage block resides at each synchronous address and on each DASD;
second means for identifying a first or a second DASD failure occurring in either the first or second arrays;
means responsive to any single DASD failure identified by the second means for processing, in the array in which the failure occurred, each of the K-K/(N+P+1) parity groups by logically combining N+P-1 blocks belonging to the group from N+P-1 other DASDs into a single block, and, writing said single block into a counterpart one of the remaining K*(N+P)/(N+P+1) spare blocks such that no two blocks of the same parity group are distributed on the same DASD; and means responsive to any second DASD failure identified in the same or other array by the second means for processing, in the array having available spare blocks, each of the K-K/(N+2) parity groups by logically combining N+P-1 blocks belonging to the group from N+P-1 other DASDs in the array in which the second failure occurred into a single block, and, writing said single block into a counterpart one of the remaining K*(N+P)/(N+P+1) spare blocks such that no two blocks of the same parity group are distributed on the same DASD.
20. The storage subsystem according to claim 19, wherein the means responsive to said second failure DASD failure maintains the subsystem in a lossless information state only if the second failure occurs after the means responsive to the first failure has rebuilt the parity groups on the failed DASD and written them into counterpart spare blocks.
21. In a storage subsystem comprising a first and a second failure independent array of DASDs, each DASD
having the capacity to store K blocks, a method for rebuilding portions of parity groups resident on a failed DASD, each parity group including N data and P parity blocks, comprising the steps of:
(a) configuring a first array of N+P+1 DASDs and a second array of N+P DASDs;
(b) distributing up to K parity groups in synchronous array addresses across subsets of N+P DASDs out of N+P+1 DASDs of the first array and K parity groups across N+P DASDs of the second array such that no two blocks from the same parity group reside on the same DASD, and distributing K blocks as spare storage across the first array such that each DASD includes K/(N+P+1) spare blocks thereon, each synchronous address and DASD
having only one spare block thereon;
(c) in the event of a single DASD failure occurring in the first array, for each of the K-K/(N+P+1) parity groups, logically combining N+P-1 blocks belonging to the group from N+P-1 other DASDs into a single block, and, writing said single block into a counterpart one of the remaining K*(N+P)/(N+P+1) spare blocks such that no two blocks of the same parity group are distributed on the same DASD; and (d) in the event of a single DASD failure occurring in the second array, for each of the K parity groups, logically combining the N+P-1 blocks belonging to the group from the N+P-1 other DASDs into a single block, and, writing said single block into a counterpart one of the K spare blocks located on the first array.
having the capacity to store K blocks, a method for rebuilding portions of parity groups resident on a failed DASD, each parity group including N data and P parity blocks, comprising the steps of:
(a) configuring a first array of N+P+1 DASDs and a second array of N+P DASDs;
(b) distributing up to K parity groups in synchronous array addresses across subsets of N+P DASDs out of N+P+1 DASDs of the first array and K parity groups across N+P DASDs of the second array such that no two blocks from the same parity group reside on the same DASD, and distributing K blocks as spare storage across the first array such that each DASD includes K/(N+P+1) spare blocks thereon, each synchronous address and DASD
having only one spare block thereon;
(c) in the event of a single DASD failure occurring in the first array, for each of the K-K/(N+P+1) parity groups, logically combining N+P-1 blocks belonging to the group from N+P-1 other DASDs into a single block, and, writing said single block into a counterpart one of the remaining K*(N+P)/(N+P+1) spare blocks such that no two blocks of the same parity group are distributed on the same DASD; and (d) in the event of a single DASD failure occurring in the second array, for each of the K parity groups, logically combining the N+P-1 blocks belonging to the group from the N+P-1 other DASDs into a single block, and, writing said single block into a counterpart one of the K spare blocks located on the first array.
22. In a storage subsystem having a plurality of DASDs, a method for rebuilding portions of parity groups resident on a failed one of said DASDs, each parity group including N data and P parity blocks, each DASD storing K
blocks, comprising the steps of:
(a) configuring 2*(N+P)+1 DASDs of the plurality to form first and second addressible arrays of N+P DASDs each with one DASD shared by both arrays;
(b) distributing K spare blocks across the 2*(N+P)+1 DASDs such that each synchronous address thereacross and each DASD includes only K/(2*(N+P)+1) blocks, and (1) distributing K parity groups across the first array of DASDs such that no synchronous first array address has more than a single parity block and that each parity group is written into a counterpart first array synchronous address to effectuate block offset or rotation from address to address; and (2) distributing K other parity groups across subsets of N+P DASDs of the N+P+1 DASDs addressibly part of the second array such that no synchronous second array address has more than a single parity block and that each said other parity group is written into a counterpart second array synchronous address to effectuate block offset or rotation from address to address; and (c) in the event of a single DASD failure occurring in either the first or second array, for each of the parity groups in that array, logically combining the remaining blocks belonging to the group from other DASDs in that array into a single block, and, writing said single block into a counterpart one of the remaining spare blocks distributed across both arrays such that no two blocks of the same parity group are distributed on the same DASD.
blocks, comprising the steps of:
(a) configuring 2*(N+P)+1 DASDs of the plurality to form first and second addressible arrays of N+P DASDs each with one DASD shared by both arrays;
(b) distributing K spare blocks across the 2*(N+P)+1 DASDs such that each synchronous address thereacross and each DASD includes only K/(2*(N+P)+1) blocks, and (1) distributing K parity groups across the first array of DASDs such that no synchronous first array address has more than a single parity block and that each parity group is written into a counterpart first array synchronous address to effectuate block offset or rotation from address to address; and (2) distributing K other parity groups across subsets of N+P DASDs of the N+P+1 DASDs addressibly part of the second array such that no synchronous second array address has more than a single parity block and that each said other parity group is written into a counterpart second array synchronous address to effectuate block offset or rotation from address to address; and (c) in the event of a single DASD failure occurring in either the first or second array, for each of the parity groups in that array, logically combining the remaining blocks belonging to the group from other DASDs in that array into a single block, and, writing said single block into a counterpart one of the remaining spare blocks distributed across both arrays such that no two blocks of the same parity group are distributed on the same DASD.
23. The storage subsystem according to any one of the claims 11, 16, or 19, wherein said first means for distributing parity groups includes means for alternatively distributing parity blocks to only a dedicated one of the DASDs.
24. The method according to any one of the claims 1, 2, or 21, wherein the step of distributing parity groups includes the step for alternatively distributing parity blocks to a dedicated one of the DASDs.
Applications Claiming Priority (2)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| US07/714,969 | 1991-06-13 | ||
| US07/714,969 US5258984A (en) | 1991-06-13 | 1991-06-13 | Method and means for distributed sparing in DASD arrays |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CA2063897A1 CA2063897A1 (en) | 1992-12-14 |
| CA2063897C true CA2063897C (en) | 1998-05-05 |
Family
ID=24872213
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CA002063897A Expired - Fee Related CA2063897C (en) | 1991-06-13 | 1992-03-24 | Method and means for distributed sparing in dasd arrays |
Country Status (9)
| Country | Link |
|---|---|
| US (1) | US5258984A (en) |
| EP (1) | EP0518603B1 (en) |
| JP (1) | JPH0724039B2 (en) |
| KR (1) | KR940011039B1 (en) |
| BR (1) | BR9202160A (en) |
| CA (1) | CA2063897C (en) |
| DE (1) | DE69219208T2 (en) |
| HK (1) | HK1000076A1 (en) |
| TW (1) | TW210386B (en) |
Families Citing this family (118)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2603757B2 (en) * | 1990-11-30 | 1997-04-23 | 富士通株式会社 | Method of controlling array disk device |
| US5537566A (en) * | 1991-12-17 | 1996-07-16 | Fujitsu Limited | Apparatus and method for controlling background processing in disk array device |
| US5469566A (en) * | 1992-03-12 | 1995-11-21 | Emc Corporation | Flexible parity generation circuit for intermittently generating a parity for a plurality of data channels in a redundant array of storage units |
| US5708668A (en) * | 1992-05-06 | 1998-01-13 | International Business Machines Corporation | Method and apparatus for operating an array of storage devices |
| JPH05341918A (en) * | 1992-05-12 | 1993-12-24 | Internatl Business Mach Corp <Ibm> | Connector for constituting duplex disk storage device system |
| US5487160A (en) * | 1992-12-04 | 1996-01-23 | At&T Global Information Solutions Company | Concurrent image backup for disk storage system |
| US5519849A (en) * | 1992-12-07 | 1996-05-21 | Digital Equipment Corporation | Method of reducing the complexity of an I/O request to a RAID-4 or RAID-5 array |
| US5579474A (en) | 1992-12-28 | 1996-11-26 | Hitachi, Ltd. | Disk array system and its control method |
| GB2278228B (en) * | 1993-05-21 | 1997-01-29 | Mitsubishi Electric Corp | An arrayed recording apparatus |
| DE69408498D1 (en) * | 1993-06-30 | 1998-03-19 | Ibm | Coding and reconstruction of the data content of up to two unavailable DASDs in a DASD arrangement |
| US5564011A (en) * | 1993-10-05 | 1996-10-08 | International Business Machines Corporation | System and method for maintaining file data access in case of dynamic critical sector failure |
| JP2912802B2 (en) * | 1993-10-14 | 1999-06-28 | 富士通株式会社 | Disk array device failure handling method and device |
| US5632012A (en) * | 1993-11-24 | 1997-05-20 | Storage Technology Corporation | Disk scrubbing system |
| US5987622A (en) * | 1993-12-10 | 1999-11-16 | Tm Patents, Lp | Parallel computer system including parallel storage subsystem including facility for correction of data in the event of failure of a storage device in parallel storage subsystem |
| US5485571A (en) * | 1993-12-23 | 1996-01-16 | International Business Machines Corporation | Method and apparatus for providing distributed sparing with uniform workload distribution in failures |
| US5537567A (en) * | 1994-03-14 | 1996-07-16 | International Business Machines Corporation | Parity block configuration in an array of storage devices |
| JP2894676B2 (en) * | 1994-03-21 | 1999-05-24 | インターナショナル・ビジネス・マシーンズ・コーポレイション | Asynchronous remote copy system and asynchronous remote copy method |
| US5657439A (en) * | 1994-08-23 | 1997-08-12 | International Business Machines Corporation | Distributed subsystem sparing |
| US5623595A (en) * | 1994-09-26 | 1997-04-22 | Oracle Corporation | Method and apparatus for transparent, real time reconstruction of corrupted data in a redundant array data storage system |
| US5615352A (en) * | 1994-10-05 | 1997-03-25 | Hewlett-Packard Company | Methods for adding storage disks to a hierarchic disk array while maintaining data availability |
| US5488701A (en) * | 1994-11-17 | 1996-01-30 | International Business Machines Corporation | In log sparing for log structured arrays |
| US5613085A (en) * | 1994-12-27 | 1997-03-18 | International Business Machines Corporation | System for parallel striping of multiple ordered data strings onto a multi-unit DASD array for improved read and write parallelism |
| US5666512A (en) * | 1995-02-10 | 1997-09-09 | Hewlett-Packard Company | Disk array having hot spare resources and methods for using hot spare resources to store user data |
| US5745915A (en) * | 1995-03-17 | 1998-04-28 | Unisys Corporation | System for parallel reading and processing of a file |
| US5657468A (en) * | 1995-08-17 | 1997-08-12 | Ambex Technologies, Inc. | Method and apparatus for improving performance in a reduntant array of independent disks |
| US5875456A (en) * | 1995-08-17 | 1999-02-23 | Nstor Corporation | Storage device array and methods for striping and unstriping data and for adding and removing disks online to/from a raid storage array |
| US5941994A (en) * | 1995-12-22 | 1999-08-24 | Lsi Logic Corporation | Technique for sharing hot spare drives among multiple subsystems |
| US5805785A (en) * | 1996-02-27 | 1998-09-08 | International Business Machines Corporation | Method for monitoring and recovery of subsystems in a distributed/clustered system |
| US6055577A (en) * | 1996-05-06 | 2000-04-25 | Oracle Corporation | System for granting bandwidth for real time processes and assigning bandwidth for non-real time processes while being forced to periodically re-arbitrate for new assigned bandwidth |
| US5862313A (en) * | 1996-05-20 | 1999-01-19 | Cray Research, Inc. | Raid system using I/O buffer segment to temporary store striped and parity data and connecting all disk drives via a single time multiplexed network |
| US5764880A (en) * | 1996-09-10 | 1998-06-09 | International Business Machines Corporation | Method and system for rebuilding log-structured arrays |
| KR100275900B1 (en) * | 1996-09-21 | 2000-12-15 | 윤종용 | Method for implement divideo parity spare disk in raid sub-system |
| US5933840A (en) * | 1997-05-19 | 1999-08-03 | International Business Machines Corporation | Garbage collection in log-structured information storage systems using age threshold selection of segments |
| US6247157B1 (en) | 1998-05-13 | 2001-06-12 | Intel Corporation | Method of encoding data signals for storage |
| US6453428B1 (en) * | 1998-07-17 | 2002-09-17 | Adaptec, Inc. | Dual-drive fault tolerant method and system for assigning data chunks to column parity sets |
| US6530035B1 (en) * | 1998-10-23 | 2003-03-04 | Oracle Corporation | Method and system for managing storage systems containing redundancy data |
| JP2000163223A (en) * | 1998-11-30 | 2000-06-16 | Hitachi Electronics Eng Co Ltd | Recording medium raid library device |
| US6532548B1 (en) * | 1999-09-21 | 2003-03-11 | Storage Technology Corporation | System and method for handling temporary errors on a redundant array of independent tapes (RAIT) |
| US6425052B1 (en) * | 1999-10-28 | 2002-07-23 | Sun Microsystems, Inc. | Load balancing configuration for storage arrays employing mirroring and striping |
| US6862692B2 (en) | 2001-01-29 | 2005-03-01 | Adaptec, Inc. | Dynamic redistribution of parity groups |
| US6990667B2 (en) | 2001-01-29 | 2006-01-24 | Adaptec, Inc. | Server-independent object positioning for load balancing drives and servers |
| US20020178162A1 (en) * | 2001-01-29 | 2002-11-28 | Ulrich Thomas R. | Integrated distributed file system with variable parity groups |
| US7054927B2 (en) * | 2001-01-29 | 2006-05-30 | Adaptec, Inc. | File system metadata describing server directory information |
| US7146524B2 (en) * | 2001-08-03 | 2006-12-05 | Isilon Systems, Inc. | Systems and methods for providing a distributed file system incorporating a virtual hot spare |
| US7685126B2 (en) | 2001-08-03 | 2010-03-23 | Isilon Systems, Inc. | System and methods for providing a distributed file system utilizing metadata to track information about data stored throughout the system |
| US6954824B2 (en) * | 2001-10-15 | 2005-10-11 | International Business Machines Corporation | Method, system, and program for determining a configuration of a logical array including a plurality of storage devices |
| US7080278B1 (en) * | 2002-03-08 | 2006-07-18 | Network Appliance, Inc. | Technique for correcting multiple storage device failures in a storage array |
| US7120826B2 (en) * | 2002-03-29 | 2006-10-10 | International Business Machines Corporation | Partial mirroring during expansion thereby eliminating the need to track the progress of stripes updated during expansion |
| US6934804B2 (en) * | 2002-05-28 | 2005-08-23 | Sun Microsystems, Inc. | Method and system for striping spares in a data storage system including an array of disk drives |
| US7085953B1 (en) | 2002-11-01 | 2006-08-01 | International Business Machines Corporation | Method and means for tolerating multiple dependent or arbitrary double disk failures in a disk array |
| US7937421B2 (en) | 2002-11-14 | 2011-05-03 | Emc Corporation | Systems and methods for restriping files in a distributed file system |
| US7062673B2 (en) * | 2003-05-19 | 2006-06-13 | Hitachi Global Technologies | System and method for sparing in RAID-1 system |
| US7143305B2 (en) * | 2003-06-25 | 2006-11-28 | International Business Machines Corporation | Using redundant spares to reduce storage device array rebuild time |
| US7533325B2 (en) * | 2003-07-14 | 2009-05-12 | International Business Machines Corporation | Anamorphic codes |
| US7254754B2 (en) | 2003-07-14 | 2007-08-07 | International Business Machines Corporation | Raid 3+3 |
| US7281177B2 (en) | 2003-07-14 | 2007-10-09 | International Business Machines Corporation | Autonomic parity exchange |
| US7328305B2 (en) | 2003-11-03 | 2008-02-05 | Network Appliance, Inc. | Dynamic parity distribution technique |
| US20050193273A1 (en) * | 2004-02-18 | 2005-09-01 | Xiotech Corporation | Method, apparatus and program storage device that provide virtual space to handle storage device failures in a storage system |
| US7237062B2 (en) * | 2004-04-02 | 2007-06-26 | Seagate Technology Llc | Storage media data structure system and method |
| GB2418769B (en) * | 2004-10-02 | 2009-06-17 | Hewlett Packard Development Co | Method and system for storing data |
| US8055711B2 (en) | 2004-10-29 | 2011-11-08 | Emc Corporation | Non-blocking commit protocol systems and methods |
| US8051425B2 (en) | 2004-10-29 | 2011-11-01 | Emc Corporation | Distributed system with asynchronous execution systems and methods |
| US8238350B2 (en) | 2004-10-29 | 2012-08-07 | Emc Corporation | Message batching with checkpoints systems and methods |
| US20070067665A1 (en) * | 2005-09-19 | 2007-03-22 | Ebrahim Hashemi | Apparatus and method for providing redundant arrays storage devices |
| US7797283B2 (en) | 2005-10-21 | 2010-09-14 | Isilon Systems, Inc. | Systems and methods for maintaining distributed data |
| US7788303B2 (en) * | 2005-10-21 | 2010-08-31 | Isilon Systems, Inc. | Systems and methods for distributed system scanning |
| US7917474B2 (en) | 2005-10-21 | 2011-03-29 | Isilon Systems, Inc. | Systems and methods for accessing and updating distributed data |
| US7346720B2 (en) * | 2005-10-21 | 2008-03-18 | Isilon Systems, Inc. | Systems and methods for managing concurrent access requests to a shared resource |
| US7386675B2 (en) * | 2005-10-21 | 2008-06-10 | Isilon Systems, Inc. | Systems and methods for using excitement values to predict future access to resources |
| US7551572B2 (en) * | 2005-10-21 | 2009-06-23 | Isilon Systems, Inc. | Systems and methods for providing variable protection |
| US7848261B2 (en) | 2006-02-17 | 2010-12-07 | Isilon Systems, Inc. | Systems and methods for providing a quiescing protocol |
| US7756898B2 (en) | 2006-03-31 | 2010-07-13 | Isilon Systems, Inc. | Systems and methods for notifying listeners of events |
| JP2008009767A (en) * | 2006-06-29 | 2008-01-17 | Hitachi Ltd | Data processing system, method thereof and storage device |
| US8539056B2 (en) | 2006-08-02 | 2013-09-17 | Emc Corporation | Systems and methods for configuring multiple network interfaces |
| US7953704B2 (en) | 2006-08-18 | 2011-05-31 | Emc Corporation | Systems and methods for a snapshot of data |
| US7822932B2 (en) | 2006-08-18 | 2010-10-26 | Isilon Systems, Inc. | Systems and methods for providing nonlinear journaling |
| US7752402B2 (en) | 2006-08-18 | 2010-07-06 | Isilon Systems, Inc. | Systems and methods for allowing incremental journaling |
| US7882071B2 (en) | 2006-08-18 | 2011-02-01 | Isilon Systems, Inc. | Systems and methods for a snapshot of data |
| US7680842B2 (en) | 2006-08-18 | 2010-03-16 | Isilon Systems, Inc. | Systems and methods for a snapshot of data |
| US7590652B2 (en) | 2006-08-18 | 2009-09-15 | Isilon Systems, Inc. | Systems and methods of reverse lookup |
| US7899800B2 (en) | 2006-08-18 | 2011-03-01 | Isilon Systems, Inc. | Systems and methods for providing nonlinear journaling |
| US7676691B2 (en) | 2006-08-18 | 2010-03-09 | Isilon Systems, Inc. | Systems and methods for providing nonlinear journaling |
| US7680836B2 (en) | 2006-08-18 | 2010-03-16 | Isilon Systems, Inc. | Systems and methods for a snapshot of data |
| US8286029B2 (en) | 2006-12-21 | 2012-10-09 | Emc Corporation | Systems and methods for managing unavailable storage devices |
| US7593938B2 (en) | 2006-12-22 | 2009-09-22 | Isilon Systems, Inc. | Systems and methods of directory entry encodings |
| US7509448B2 (en) | 2007-01-05 | 2009-03-24 | Isilon Systems, Inc. | Systems and methods for managing semantic locks |
| US7779048B2 (en) | 2007-04-13 | 2010-08-17 | Isilon Systems, Inc. | Systems and methods of providing possible value ranges |
| US8966080B2 (en) | 2007-04-13 | 2015-02-24 | Emc Corporation | Systems and methods of managing resource utilization on a threaded computer system |
| US7900015B2 (en) | 2007-04-13 | 2011-03-01 | Isilon Systems, Inc. | Systems and methods of quota accounting |
| US7882068B2 (en) | 2007-08-21 | 2011-02-01 | Isilon Systems, Inc. | Systems and methods for adaptive copy on write |
| US7949692B2 (en) | 2007-08-21 | 2011-05-24 | Emc Corporation | Systems and methods for portals into snapshot data |
| US7966289B2 (en) | 2007-08-21 | 2011-06-21 | Emc Corporation | Systems and methods for reading objects in a file system |
| US8020032B2 (en) * | 2007-12-27 | 2011-09-13 | International Business Machines Corporation | Method for providing deferred maintenance on storage subsystems |
| US7953709B2 (en) | 2008-03-27 | 2011-05-31 | Emc Corporation | Systems and methods for a read only mode for a portion of a storage system |
| US7870345B2 (en) | 2008-03-27 | 2011-01-11 | Isilon Systems, Inc. | Systems and methods for managing stalled storage devices |
| US7984324B2 (en) | 2008-03-27 | 2011-07-19 | Emc Corporation | Systems and methods for managing stalled storage devices |
| US7949636B2 (en) | 2008-03-27 | 2011-05-24 | Emc Corporation | Systems and methods for a read only mode for a portion of a storage system |
| US8019728B2 (en) * | 2008-04-17 | 2011-09-13 | Nec Laboratories America, Inc. | Dynamically quantifying and improving the reliability of distributed data storage systems |
| US8099623B1 (en) * | 2008-10-08 | 2012-01-17 | Netapp, Inc. | Efficient distributed hot sparing scheme in a parity declustered RAID organization |
| US8799743B2 (en) | 2008-10-28 | 2014-08-05 | Micron Technology, Inc. | Error correction in multiple semiconductor memory units |
| US8065558B2 (en) * | 2009-03-24 | 2011-11-22 | Lsi Corporation | Data volume rebuilder and methods for arranging data volumes for improved RAID reconstruction performance |
| JP5169993B2 (en) * | 2009-05-27 | 2013-03-27 | 日本電気株式会社 | Data storage system and data area management method |
| JP5391993B2 (en) * | 2009-10-19 | 2014-01-15 | 富士通株式会社 | Disk array device |
| US20120010942A1 (en) * | 2010-07-07 | 2012-01-12 | Yahoo! Inc. | Online advertising marketplace data provider assessment and recommendation |
| US8775733B2 (en) * | 2011-08-30 | 2014-07-08 | Hitachi, Ltd. | Distribution design for fast raid rebuild architecture based on load to limit number of redundant storage devices |
| US9563509B2 (en) | 2014-07-15 | 2017-02-07 | Nimble Storage, Inc. | Methods and systems for storing data in a redundant manner on a plurality of storage units of a storage system |
| JP6515458B2 (en) * | 2014-08-08 | 2019-05-22 | 富士通株式会社 | Storage control device, storage control program, and storage control method |
| WO2016142998A1 (en) * | 2015-03-06 | 2016-09-15 | 株式会社日立製作所 | Computer system |
| US9875052B2 (en) | 2016-03-15 | 2018-01-23 | International Business Machines Corporation | Storage capacity allocation using distributed spare space |
| US20170337103A1 (en) * | 2016-05-19 | 2017-11-23 | Intel Corporation | Method and apparatus to provide data redundancy in a solid-state drive |
| US10133630B2 (en) | 2016-09-06 | 2018-11-20 | International Business Machines Corporation | Disposable subset parities for use in a distributed RAID |
| US10459807B2 (en) * | 2017-05-23 | 2019-10-29 | International Business Machines Corporation | Determining modified portions of a RAID storage array |
| US11340789B2 (en) * | 2020-10-13 | 2022-05-24 | Dell Products L.P. | Predictive redistribution of capacity in a flexible RAID system |
| US11327668B1 (en) * | 2021-01-14 | 2022-05-10 | Dell Products L.P. | Predictable member assignment for expanding flexible raid system |
| US11144396B1 (en) * | 2021-01-27 | 2021-10-12 | Dell Products L.P. | Raid reliability with a provisional spare disk |
| US11829269B2 (en) * | 2021-01-28 | 2023-11-28 | EMC IP Holding Company LLC | Storage array disk recovery |
| WO2023023223A1 (en) * | 2021-08-20 | 2023-02-23 | Pure Storage, Inc. | Efficient partitioning for storage system resiliency groups |
| CN113986149B (en) * | 2021-12-27 | 2022-04-22 | 苏州浪潮智能科技有限公司 | System fault processing method, device, equipment and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US5195100A (en) * | 1990-03-02 | 1993-03-16 | Micro Technology, Inc. | Non-volatile memory storage of write operation identifier in data sotrage device |
| US5208813A (en) * | 1990-10-23 | 1993-05-04 | Array Technology Corporation | On-line reconstruction of a failed redundant array system |
-
1991
- 1991-06-13 US US07/714,969 patent/US5258984A/en not_active Expired - Lifetime
-
1992
- 1992-01-20 TW TW081100377A patent/TW210386B/zh active
- 1992-03-24 CA CA002063897A patent/CA2063897C/en not_active Expired - Fee Related
- 1992-04-01 JP JP4080039A patent/JPH0724039B2/en not_active Expired - Lifetime
- 1992-05-13 KR KR1019920008032A patent/KR940011039B1/en not_active IP Right Cessation
- 1992-06-05 BR BR929202160A patent/BR9202160A/en not_active Application Discontinuation
- 1992-06-08 EP EP92305255A patent/EP0518603B1/en not_active Expired - Lifetime
- 1992-06-08 DE DE69219208T patent/DE69219208T2/en not_active Expired - Lifetime
-
1997
- 1997-07-26 HK HK97101607A patent/HK1000076A1/en not_active IP Right Cessation
Also Published As
| Publication number | Publication date |
|---|---|
| EP0518603A2 (en) | 1992-12-16 |
| EP0518603B1 (en) | 1997-04-23 |
| DE69219208D1 (en) | 1997-05-28 |
| DE69219208T2 (en) | 1997-10-16 |
| EP0518603A3 (en) | 1993-10-06 |
| JPH05100960A (en) | 1993-04-23 |
| TW210386B (en) | 1993-08-01 |
| US5258984A (en) | 1993-11-02 |
| BR9202160A (en) | 1993-02-02 |
| HK1000076A1 (en) | 1997-11-14 |
| KR940011039B1 (en) | 1994-11-22 |
| JPH0724039B2 (en) | 1995-03-15 |
| CA2063897A1 (en) | 1992-12-14 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CA2063897C (en) | Method and means for distributed sparing in dasd arrays | |
| US5301297A (en) | Method and means for managing RAID 5 DASD arrays having RAID DASD arrays as logical devices thereof | |
| US5889795A (en) | Disk array system and method for storing data | |
| US5488701A (en) | In log sparing for log structured arrays | |
| EP0485110B1 (en) | Logical partitioning of a redundant array storage system | |
| US5303244A (en) | Fault tolerant disk drive matrix | |
| US5596709A (en) | Method and apparatus for recovering parity protected data | |
| US5404361A (en) | Method and apparatus for ensuring data integrity in a dynamically mapped data storage subsystem | |
| JP3505093B2 (en) | File management system | |
| US5124987A (en) | Logical track write scheduling system for a parallel disk drive array data storage subsystem | |
| US5689678A (en) | Distributed storage array system having a plurality of modular control units | |
| US7831768B2 (en) | Method and apparatus for writing data to a disk array | |
| US5650969A (en) | Disk array system and method for storing data | |
| US20030115437A1 (en) | Disk array apparatus and data update method therein | |
| JP2000511318A (en) | Transformable RAID for Hierarchical Storage Management System | |
| AU1416092A (en) | Disk drive array memory system using nonuniform disk drives | |
| EP0515499A4 (en) | ||
| JPH06504863A (en) | Storage array with copyback cache | |
| US7240237B2 (en) | Method and system for high bandwidth fault tolerance in a storage subsystem | |
| JP2857288B2 (en) | Disk array device | |
| US5737741A (en) | Disk array with parity data storage on the radially inner part of each disk | |
| JP2857289B2 (en) | Disk array device | |
| JPH06119120A (en) | Disk array control system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| EEER | Examination request | ||
| MKLA | Lapsed |