CA2363726A1 - Methods and systems for implementing shared disk array management functions - Google Patents
Methods and systems for implementing shared disk array management functions Download PDFInfo
- Publication number
- CA2363726A1 CA2363726A1 CA002363726A CA2363726A CA2363726A1 CA 2363726 A1 CA2363726 A1 CA 2363726A1 CA 002363726 A CA002363726 A CA 002363726A CA 2363726 A CA2363726 A CA 2363726A CA 2363726 A1 CA2363726 A1 CA 2363726A1
- Authority
- CA
- Canada
- Prior art keywords
- amfs
- resource
- amf
- lock
- redundancy group
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0646—Horizontal data movement in storage systems, i.e. moving data in between storage devices or systems
- G06F3/065—Replication mechanisms
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1076—Parity data used in redundant arrays of independent storages, e.g. in RAID systems
- G06F11/1088—Reconstruction on already foreseen single or plurality of spare disks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/08—Error detection or correction by redundancy in data representation, e.g. by using checking codes
- G06F11/10—Adding special bits or symbols to the coded information, e.g. parity check, casting out 9's or 11's
- G06F11/1076—Parity data used in redundant arrays of independent storages, e.g. in RAID systems
- G06F11/1096—Parity calculation or recalculation after configuration or reconfiguration of the system
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
- G06F11/2089—Redundant storage control functionality
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2097—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements maintaining the standby controller/processing unit updated
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0604—Improving or facilitating administration, e.g. storage management
- G06F3/0607—Improving or facilitating administration, e.g. storage management by facilitating the process of upgrading existing storage systems, e.g. for improving compatibility between host and storage device
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/061—Improving I/O performance
- G06F3/0613—Improving I/O performance in relation to throughput
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0602—Interfaces specially adapted for storage systems specifically adapted to achieve a particular effect
- G06F3/0614—Improving the reliability of storage systems
- G06F3/0619—Improving the reliability of storage systems in relation to data integrity, e.g. data losses, bit errors
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0629—Configuration or reconfiguration of storage systems
- G06F3/0632—Configuration or reconfiguration of storage systems by initialisation or re-initialisation of storage systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/067—Distributed or networked storage systems, e.g. storage area networks [SAN], network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0668—Interfaces specially adapted for storage systems adopting a particular infrastructure
- G06F3/0671—In-line storage system
- G06F3/0683—Plurality of storage devices
- G06F3/0689—Disk arrays, e.g. RAID, JBOD
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
- G06F9/526—Mutual exclusion algorithms
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
- H04L67/1097—Protocols in which an application is distributed across nodes in the network for distributed storage of data in networks, e.g. transport arrangements for network file system [NFS], storage area networks [SAN] or network attached storage [NAS]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/16—Error detection or correction of the data by redundancy in hardware
- G06F11/20—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements
- G06F11/2053—Error detection or correction of the data by redundancy in hardware using active fault-masking, e.g. by switching out faulty elements or by switching in spare elements where persistent mass storage functionality or persistent mass storage control functionality is redundant
- G06F11/2094—Redundant storage or storage space
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
- G06F2201/85—Active fault masking without idle spares
Abstract
Multiple Array Management Functions (80) included in controller (30) are connected to multiple redundancy groups (40) over a storage area network (SAN), such as a fiber-channel based SAN (50). The multiple Array Management Functions share management responsibility of the redundancy groups, each of which typically includes multiple resources spread over multiple disks (45). The AMFs provide concurrent access to the redundancy groups for associated host systems. When a host requests an AMF to perform an operation on a resource, the AMF synchronizes with the other AMFs sharing control of the redundancy group that includes the resource to be operated on, so as to obta in a lock on the resource. While performing the operation, the AMF send replication data and state information associated with the resource such tha t if the AMF fails, any of the other AMFs are able to complete the operation a nd maintain data reliability and coherency.
Claims (38)
1. A data storage system comprising:
a redundancy group including a plurality of resources;
two or more array management functions (AMFs) sharing access to the redundancy group, wherein the AMFs provide concurrent access to the redundancy group for associated host systems; and an interconnect medium for connecting the AMFs with the redundancy group;
wherein when a first one of the AMFs desires to perform an operation on a first resource in the redundancy group, the first AMF arbitrates with the other AMFs sharing access to the redundancy group for a lock on the first resource, whereupon the first AMF performs the operation on the first resource and concurrently sends replication data and state information associated with the first resource to the other AMFs such that if the first AMF fails while performing the operation, one of the other AMFs is able to complete the operation.
a redundancy group including a plurality of resources;
two or more array management functions (AMFs) sharing access to the redundancy group, wherein the AMFs provide concurrent access to the redundancy group for associated host systems; and an interconnect medium for connecting the AMFs with the redundancy group;
wherein when a first one of the AMFs desires to perform an operation on a first resource in the redundancy group, the first AMF arbitrates with the other AMFs sharing access to the redundancy group for a lock on the first resource, whereupon the first AMF performs the operation on the first resource and concurrently sends replication data and state information associated with the first resource to the other AMFs such that if the first AMF fails while performing the operation, one of the other AMFs is able to complete the operation.
2. The data storage system of claim 1, wherein the first AMF does not release the lock on the first resource until a second one of the AMFs arbitrates for a lock on the first resource.
3. The data storage system of claim 1, wherein if the first AMF fails, the remaining AMFs arbitrate for a lock on the first resource, whereupon a second one of the AMFs obtains the lock and completes the operation.
4. The data storage system of claim 1, wherein the operation performed by the first AMF on the first resource includes a plurality of steps, wherein the first AMF performs each step of the operation on the resource, and for each step concurrently sends replication data and state information associated with the first resource to the remaining AMFs, such that if the first AMF fails while performing any of the steps of the operation, one of the remaining AMFs is able to complete the operation.
5. The data storage system of claim 1, wherein one or more host systems communicate with the first AMF over one of the interconnect medium, an Internet connection and a PCI bus.
6. The system of claim 1, wherein the interconnect medium includes at least one of a SCSI interface, a fiber-channel interface, a storage area network and a fiber-channel storage area network.
7. The system of claim 1, wherein each of the AMFs executes in one of a controller, a host bus adapter and a host computer.
8. In a data storage system, a method of dynamically sharing management of a redundancy group between two or more array management functions (AMFs), wherein the AMFs are able to concurrently access the redundancy group, the redundancy group including a plurality of resources, the method comprising the steps of:
receiving a request from a host by a first one of the AMFs to perform a first operation on a first one of the resources;
synchronizing with the other AMFs so as-to acquire access to the first resource; and performing the first operation on the first resource.
receiving a request from a host by a first one of the AMFs to perform a first operation on a first one of the resources;
synchronizing with the other AMFs so as-to acquire access to the first resource; and performing the first operation on the first resource.
9. The method of claim 8, wherein the first operation is a read operation, the method further comprising the steps of:
receiving a request from a host by a second one of the AMFs to perform a second read operation on the first resource; and performing the second read operation on the first resource by the second AMF concurrently with the first AMF performing the first read operation on the first resource.
receiving a request from a host by a second one of the AMFs to perform a second read operation on the first resource; and performing the second read operation on the first resource by the second AMF concurrently with the first AMF performing the first read operation on the first resource.
10. The method of claim 8, wherein the step of synchronizing includes the step of arbitrating with the one or more other AMFs to acquire a lock on the first resource so that no other AMF is able to access the first resource while the first AMF
maintains the lock.
maintains the lock.
11. The method of claim 10, further comprising the step of releasing the lock on the first resource only when another AMF arbitrates for the lock.
12. The method of claim 8, further comprising the step of sending replication data and state information associated with the first resource to the other AMFs, concurrently with the step of performing the first operation, such that if the first AMF fails, one of the other AMFs is able to complete the first operation.
13. The method of claim 12, wherein if the first AMF fails while performing the first operation, the method further comprises the step of completing the first operation with a second one of the AMFs.
14. The method of claim 13, wherein the step of completing the first operation comprises the step of the second AMF synchronizing with the other AMFs so as to acquire access to the first resource.
15. The method of claim 8, wherein the first operation includes a plurality of sub-operations, wherein the step of performing the first operation includes performing each sub-operation on the first resource, the method further comprising sending replication data and state information associated with the first resource to the other AMFs concurrently with the performance of each sub-operation, such that if the first AMF fails, one of the other AMFs is able to complete the first operation.
16. The method of claim 8, further comprising the steps of:
receiving a request from a second host by a second one of the AMFs to perform a second operation on a second one of the resources;
synchronizing with the first AMF and the other AMFs so as to acquire access to the second resource; and performing the second operation on the second resource.
receiving a request from a second host by a second one of the AMFs to perform a second operation on a second one of the resources;
synchronizing with the first AMF and the other AMFs so as to acquire access to the second resource; and performing the second operation on the second resource.
17. The method of claim 16, wherein at least a portion of the second operation is performed concurrently with the performance of the first operation.
18. A data storage network system comprising:
one or more redundancy groups, each redundancy group including multiple resources spread over multiple disks;
two or more array management functions (AMFs) sharing redundancy group management of the one or more redundancy groups, wherein the AMFs are able to concurrently access the one or more redundancy groups; and a interconnect medium for interconnecting the AMFs with the one or more redundancy groups.
one or more redundancy groups, each redundancy group including multiple resources spread over multiple disks;
two or more array management functions (AMFs) sharing redundancy group management of the one or more redundancy groups, wherein the AMFs are able to concurrently access the one or more redundancy groups; and a interconnect medium for interconnecting the AMFs with the one or more redundancy groups.
19. The system of claim 18, wherein a first one of the redundancy groups includes a replacement disk, and wherein if one of the disks in the first redundancy group fails, at least two of the AMFs each arbitrate for control of one or more of the resources on the first redundancy group, such that each of the at least two AMFs are able to concurrently reconstruct the first redundancy group using the replacement disk.
20. The system of claim 18, wherein if an extra disk is added to a first one of the redundancy groups, at least two of the AMFs each arbitrate for control of one or more of the resources on the first redundancy group, such that each of the at least two AMFs are able to concurrently expand the redundancy group using the extra disk.
21. The system of claim 18, wherein a first one of the AMFs receives a write command from a host to write at least two data sets to two or more of the resources, wherein the first AMF acquires a lock on the first resource to which the first data set is to be written, writes the first data set to the first resource and concurrently performs a replication operation wherein replication data and state information associated with the first resource is sent to the other AMFs, such that if the first AMF fails while performing the write operation, one of the other AMFs is able to complete the write operation.
22. The system of claim 21, wherein concurrently with sending replication data and state information, the first AMF determines if the second data set is to be written to a second resource, and if so acquires a lock on the second resource, writes the second data set to the second resource and concurrently performs a second replication operation wherein replication data and state information associated with the second resource is sent to the other AMFs, whereupon the first AMF waits until all replication operations have completed before sending status information to the host.
23. The system of claim 18, wherein each of the AMFs includes a means for pipelining replication of incoming host data.
24. The system of claim 18, further comprising two or more controllers, each controller implementing at least one of the two or more AMFs, wherein each of the AMFs communicates with each other over one of the interconnect medium and one or more PCI busses, and wherein the system further includes a means for gathering messages, wherein many small messages destined for a particular one of the controllers are combined and sent as one message to the particular controller.
25. The system of claim 18, wherein the interconnect medium includes at least one of a SCSI interface, a fiber-channel interface, a storage area network and a fiber-channel storage area network.
26. A method of reconstructing a redundancy group when one of its disks fails in a data storage network system comprising two or more array management functions (AMFs) interconnected with the redundancy group, wherein the redundancy group includes multiple resources spread over multiple disks, wherein the redundancy group also includes a replacement disk, wherein the AMFs all share management of the redundancy group, and wherein the AMFs are able to concurrently access the redundancy group, the method comprising the steps of:
arbitrating for control of a first resource by a first one of the AMFs;
arbitrating for control of a second resource by a second one of the AMFs;
and concurrently reconstructing the first and second resources using the replacement disk.
arbitrating for control of a first resource by a first one of the AMFs;
arbitrating for control of a second resource by a second one of the AMFs;
and concurrently reconstructing the first and second resources using the replacement disk.
27. A method of expanding a redundancy group when an extra disk is added to it in a data storage network system comprising two or more array management functions (AMFs) interconnected with the redundancy group, wherein the redundancy group includes multiple resources spread over multiple disks, wherein the AMFs all share management of the redundancy group, and wherein the AMFs are able to concurrently access the redundancy group, the method comprising the steps of:
arbitrating for control of a first resource by a first one of the AMFs;
arbitrating for control of a second resource by a second one of the AMFs;
and concurrently expanding the first and second resources using the extra disk.
arbitrating for control of a first resource by a first one of the AMFs;
arbitrating for control of a second resource by a second one of the AMFs;
and concurrently expanding the first and second resources using the extra disk.
28. A method of pipelining replication of incoming host data in a data storage network system comprising a redundancy group interconnected with two or more array management functions (AMFs), wherein the redundancy group includes multiple resources spread over multiple disks, wherein the AMFs all share management of the redundancy group, and wherein the AMFs are able to concurrently access the redundancy group, the method comprising the steps of:
receiving a write command by a first AMF from the host to write at least two data sets to two or more of the resources;
acquiring a lock by the first AMF on the first resource to which the first data set is to be written;
writing the first data set to the first resource; and concurrently performing a first replication operation wherein replication data and state information associated with the first resource is sent to the other AMFs, such that if the first AMF fails while performing the write operation, one of the other AMFs is able to complete the write operation.
receiving a write command by a first AMF from the host to write at least two data sets to two or more of the resources;
acquiring a lock by the first AMF on the first resource to which the first data set is to be written;
writing the first data set to the first resource; and concurrently performing a first replication operation wherein replication data and state information associated with the first resource is sent to the other AMFs, such that if the first AMF fails while performing the write operation, one of the other AMFs is able to complete the write operation.
29. The method of claim 28, further comprising the steps of:
concurrently with performing the first replication operation, determining if the second data set is to be written to a second resource, and if so acquiring a lock on the second resource;
writing the second data set to the second resource; and concurrently performing a second replication operation wherein replication data and state information associated with the second resource is sent to the other AMFs; and thereafter sending status information to the host after all replication operations have completed.
concurrently with performing the first replication operation, determining if the second data set is to be written to a second resource, and if so acquiring a lock on the second resource;
writing the second data set to the second resource; and concurrently performing a second replication operation wherein replication data and state information associated with the second resource is sent to the other AMFs; and thereafter sending status information to the host after all replication operations have completed.
30. In a data storage system, a method of dynamically sharing management of a redundancy group between two or more array management functions (AMFs), wherein the AMFs are able to concurrently access the redundancy group, the redundancy group including a plurality of resources, the method comprising the steps of:
determining an arbiter AMF for a first one of the resources, wherein the arbiter AMF is one of the two or more AMFs sharing management of the redundancy group, and wherein the arbiter AMF is able to grant a lock for the first resource;
communicating a lock request from a first one of the AMFs to the arbiter AMF requesting a lock on the first resource; and performing an operation on the first resource by the first AMF once the lock on the first resource has been granted by the arbiter AMF.
determining an arbiter AMF for a first one of the resources, wherein the arbiter AMF is one of the two or more AMFs sharing management of the redundancy group, and wherein the arbiter AMF is able to grant a lock for the first resource;
communicating a lock request from a first one of the AMFs to the arbiter AMF requesting a lock on the first resource; and performing an operation on the first resource by the first AMF once the lock on the first resource has been granted by the arbiter AMF.
31. The method of claim 30, wherein a second AMF has a lock on the first resource, the method further comprising the steps of:
issuing a release lock request to the second AMF from the arbiter AMF;
communicating a lock-released message from the second AMF to the arbiter AMF; and thereafter granting a lock on the first resource to the first AMF so that the first AMF
is able to perform the operation.
issuing a release lock request to the second AMF from the arbiter AMF;
communicating a lock-released message from the second AMF to the arbiter AMF; and thereafter granting a lock on the first resource to the first AMF so that the first AMF
is able to perform the operation.
32. The method of claim 30, wherein no other AMF has a lock on the first resource when the first AMF communicates the lock request, the method further comprising the steps of:
immediately granting a lock on the first resource to the first AMF so that the first AMF is able to perform the operation.
immediately granting a lock on the first resource to the first AMF so that the first AMF is able to perform the operation.
33. The method of claim 30, wherein the lock requested by the first AMF is a write lock, and wherein once the lock is granted, no other AMF is able to obtain a lock on the first resource until the first AMF releases the write lock.
34. The method of claim 30, wherein the lock requested by the first AMF is a read lock, and wherein any of the other AMFs is able to concurrently obtain a read lock on the first resource such that multiple AMFs may read the first resource concurrently.
35 35. The method of claim 30, wherein the step of determining the arbiter AMF includes assigning one of the two or more AMFs to be the arbiter AMF for the first resource based on at least one of the number of the AMFs sharing the resource and the location of the resource.
36. The method of claim 35, further including the step of reassigning a different one of the two or more AMFs to be the arbiter AMF for the first resource if the original arbiter AMF fails.
37. The method of claim 30, wherein the step of determining the arbiter AMF includes assigning the AMF that last had a write lock on the first resource to be the arbiter AMF.
38. The method of claim 30, wherein the arbiter AMF is able to grant a lock on one or more prefetch resources, wherein the method further includes the steps of:
specifying a first one of the prefetch resources by the first AMF, concurrently with requesting a lock on the first resource; and granting a lock on the first prefetch resource to the first AMF if the first prefetch is not already locked, such that the first AMF will already have the lock on the first prefetch resource when it later requests the lock.
specifying a first one of the prefetch resources by the first AMF, concurrently with requesting a lock on the first resource; and granting a lock on the first prefetch resource to the first AMF if the first prefetch is not already locked, such that the first AMF will already have the lock on the first prefetch resource when it later requests the lock.
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US09/261,906 | 1999-03-03 | ||
US09/261,906 US6148414A (en) | 1998-09-24 | 1999-03-03 | Methods and systems for implementing shared disk array management functions |
PCT/US2000/003275 WO2000052576A1 (en) | 1999-03-03 | 2000-02-08 | Methods and systems for implementing shared disk array management functions |
Publications (2)
Publication Number | Publication Date |
---|---|
CA2363726A1 true CA2363726A1 (en) | 2000-09-08 |
CA2363726C CA2363726C (en) | 2010-06-29 |
Family
ID=22995391
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
CA2363726A Expired - Lifetime CA2363726C (en) | 1999-03-03 | 2000-02-08 | Methods and systems for implementing shared disk array management functions |
Country Status (12)
Country | Link |
---|---|
US (3) | US6148414A (en) |
EP (2) | EP1796001B1 (en) |
JP (1) | JP2002538549A (en) |
KR (1) | KR20020012539A (en) |
CN (1) | CN100489796C (en) |
AT (2) | ATE359552T1 (en) |
AU (1) | AU3358600A (en) |
CA (1) | CA2363726C (en) |
DE (2) | DE60042225D1 (en) |
NZ (1) | NZ513789A (en) |
TW (1) | TW468107B (en) |
WO (1) | WO2000052576A1 (en) |
Families Citing this family (243)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7389312B2 (en) * | 1997-04-28 | 2008-06-17 | Emc Corporation | Mirroring network data to establish virtual storage area network |
JP3360719B2 (en) * | 1998-06-19 | 2002-12-24 | 日本電気株式会社 | Disk array clustering reporting method and system |
US6148414A (en) * | 1998-09-24 | 2000-11-14 | Seek Systems, Inc. | Methods and systems for implementing shared disk array management functions |
JP2000187561A (en) * | 1998-12-24 | 2000-07-04 | Hitachi Ltd | Storage device system |
JP4294142B2 (en) * | 1999-02-02 | 2009-07-08 | 株式会社日立製作所 | Disk subsystem |
US6397350B1 (en) * | 1999-02-19 | 2002-05-28 | International Business Machines Corporation | Method of providing direct data processing access using a queued direct input-output device |
US6449731B1 (en) * | 1999-03-03 | 2002-09-10 | Tricord Systems, Inc. | Self-healing computer system storage |
US7266706B2 (en) * | 1999-03-03 | 2007-09-04 | Yottayotta, Inc. | Methods and systems for implementing shared disk array management functions |
US6400730B1 (en) * | 1999-03-10 | 2002-06-04 | Nishan Systems, Inc. | Method and apparatus for transferring data between IP network devices and SCSI and fibre channel devices over an IP network |
US6446141B1 (en) | 1999-03-25 | 2002-09-03 | Dell Products, L.P. | Storage server system including ranking of data source |
US6553408B1 (en) * | 1999-03-25 | 2003-04-22 | Dell Products L.P. | Virtual device architecture having memory for storing lists of driver modules |
US6640278B1 (en) * | 1999-03-25 | 2003-10-28 | Dell Products L.P. | Method for configuration and management of storage resources in a storage network |
US6654830B1 (en) | 1999-03-25 | 2003-11-25 | Dell Products L.P. | Method and system for managing data migration for a storage system |
IE20000203A1 (en) * | 1999-03-25 | 2001-02-21 | Converge Net Technologies Inc | Storage domain management system |
US6341328B1 (en) * | 1999-04-20 | 2002-01-22 | Lucent Technologies, Inc. | Method and apparatus for using multiple co-dependent DMA controllers to provide a single set of read and write commands |
JP4461511B2 (en) * | 1999-06-09 | 2010-05-12 | 株式会社日立製作所 | Disk array device and data read / write method to disk device |
US6519679B2 (en) * | 1999-06-11 | 2003-02-11 | Dell Usa, L.P. | Policy based storage configuration |
JP2001034427A (en) * | 1999-07-23 | 2001-02-09 | Fujitsu Ltd | Device controller and its method |
US6961749B1 (en) * | 1999-08-25 | 2005-11-01 | Network Appliance, Inc. | Scalable file server with highly available pairs |
US6499058B1 (en) * | 1999-09-09 | 2002-12-24 | Motokazu Hozumi | File shared apparatus and its method file processing apparatus and its method recording medium in which file shared program is recorded and recording medium in which file processing program is recorded |
US6343324B1 (en) * | 1999-09-13 | 2002-01-29 | International Business Machines Corporation | Method and system for controlling access share storage devices in a network environment by configuring host-to-volume mapping data structures in the controller memory for granting and denying access to the devices |
US8165146B1 (en) | 1999-10-28 | 2012-04-24 | Lightwaves Systems Inc. | System and method for storing/caching, searching for, and accessing data |
US6944654B1 (en) * | 1999-11-01 | 2005-09-13 | Emc Corporation | Multiple storage array control |
US6351776B1 (en) | 1999-11-04 | 2002-02-26 | Xdrive, Inc. | Shared internet storage resource, user interface system, and method |
US20100185614A1 (en) | 1999-11-04 | 2010-07-22 | O'brien Brett | Shared Internet storage resource, user interface system, and method |
US6721900B1 (en) * | 1999-12-22 | 2004-04-13 | Rockwell Automation Technologies, Inc. | Safety network for industrial controller having reduced bandwidth requirements |
US6834326B1 (en) * | 2000-02-04 | 2004-12-21 | 3Com Corporation | RAID method and device with network protocol between controller and storage devices |
US6877044B2 (en) | 2000-02-10 | 2005-04-05 | Vicom Systems, Inc. | Distributed storage management platform architecture |
US6772270B1 (en) * | 2000-02-10 | 2004-08-03 | Vicom Systems, Inc. | Multi-port fibre channel controller |
US6826711B2 (en) * | 2000-02-18 | 2004-11-30 | Avamar Technologies, Inc. | System and method for data protection with multidimensional parity |
US20010044879A1 (en) * | 2000-02-18 | 2001-11-22 | Moulton Gregory Hagan | System and method for distributed management of data storage |
US7509420B2 (en) * | 2000-02-18 | 2009-03-24 | Emc Corporation | System and method for intelligent, globally distributed network storage |
US7299290B2 (en) * | 2000-03-22 | 2007-11-20 | Yottayotta, Inc. | Method and system for providing multimedia information on demand over wide area networks |
US6490659B1 (en) * | 2000-03-31 | 2002-12-03 | International Business Machines Corporation | Warm start cache recovery in a dual active controller with cache coherency using stripe locks for implied storage volume reservations |
US6754718B1 (en) | 2000-05-10 | 2004-06-22 | Emc Corporation | Pushing attribute information to storage devices for network topology access |
US6950871B1 (en) * | 2000-06-29 | 2005-09-27 | Hitachi, Ltd. | Computer system having a storage area network and method of handling data in the computer system |
US6625747B1 (en) * | 2000-06-30 | 2003-09-23 | Dell Products L.P. | Computer storage system and failover method |
US7188157B1 (en) * | 2000-06-30 | 2007-03-06 | Hitachi, Ltd. | Continuous update of data in a data server system |
US6820171B1 (en) * | 2000-06-30 | 2004-11-16 | Lsi Logic Corporation | Methods and structures for an extensible RAID storage architecture |
US7281032B2 (en) * | 2000-06-30 | 2007-10-09 | Hitachi, Ltd. | File sharing system with data mirroring by storage systems |
US6928470B1 (en) * | 2000-07-31 | 2005-08-09 | Western Digital Ventures, Inc. | Transferring scheduling data from a plurality of disk storage devices to a network switch before transferring data associated with scheduled requests between the network switch and a plurality of host initiators |
JP2004506980A (en) * | 2000-08-11 | 2004-03-04 | 3ウェア、 インコーポレイテッド | Architecture for providing block-level storage access over a computer network |
US6603625B1 (en) * | 2000-09-11 | 2003-08-05 | Western Digital Technologies, Inc. | Spindle synchronizing a plurality of spindles connected to a multi-dimensional computer network |
US6721902B1 (en) * | 2000-10-12 | 2004-04-13 | Hewlett-Packard Development Company, L.P. | Method and system for providing LUN-based backup reliability via LUN-based locking |
US6810491B1 (en) * | 2000-10-12 | 2004-10-26 | Hitachi America, Ltd. | Method and apparatus for the takeover of primary volume in multiple volume mirroring |
US6901451B1 (en) * | 2000-10-31 | 2005-05-31 | Fujitsu Limited | PCI bridge over network |
US6671773B2 (en) * | 2000-12-07 | 2003-12-30 | Spinnaker Networks, Llc | Method and system for responding to file system requests |
WO2002065275A1 (en) * | 2001-01-11 | 2002-08-22 | Yottayotta, Inc. | Storage virtualization system and methods |
US6907457B2 (en) * | 2001-01-25 | 2005-06-14 | Dell Inc. | Architecture for access to embedded files using a SAN intermediate device |
US6990547B2 (en) * | 2001-01-29 | 2006-01-24 | Adaptec, Inc. | Replacing file system processors by hot swapping |
US20020191311A1 (en) * | 2001-01-29 | 2002-12-19 | Ulrich Thomas R. | Dynamically scalable disk array |
US20020138559A1 (en) * | 2001-01-29 | 2002-09-26 | Ulrich Thomas R. | Dynamically distributed file system |
US20020156973A1 (en) | 2001-01-29 | 2002-10-24 | Ulrich Thomas R. | Enhanced disk array |
US7054927B2 (en) | 2001-01-29 | 2006-05-30 | Adaptec, Inc. | File system metadata describing server directory information |
US6862692B2 (en) | 2001-01-29 | 2005-03-01 | Adaptec, Inc. | Dynamic redistribution of parity groups |
US6990667B2 (en) | 2001-01-29 | 2006-01-24 | Adaptec, Inc. | Server-independent object positioning for load balancing drives and servers |
US6904544B2 (en) | 2001-01-30 | 2005-06-07 | Sun Microsystems, Inc. | Method, system, program, and data structures for testing a network system including input/output devices |
EP1370945B1 (en) * | 2001-02-13 | 2010-09-08 | Candera, Inc. | Failover processing in a storage system |
IES20010611A2 (en) * | 2001-03-08 | 2002-09-18 | Richmount Computers Ltd | Distributed lock management chip |
US8766773B2 (en) * | 2001-03-20 | 2014-07-01 | Lightwaves Systems, Inc. | Ultra wideband radio frequency identification system, method, and apparatus |
US8270452B2 (en) * | 2002-04-30 | 2012-09-18 | Lightwaves Systems, Inc. | Method and apparatus for multi-band UWB communications |
US7545868B2 (en) * | 2001-03-20 | 2009-06-09 | Lightwaves Systems, Inc. | High bandwidth data transport system |
US7533132B2 (en) * | 2001-03-21 | 2009-05-12 | Sap Ag | Parallel replication mechanism for state information produced by serialized processing |
US7231430B2 (en) * | 2001-04-20 | 2007-06-12 | Egenera, Inc. | Reconfigurable, virtual processing system, cluster, network and method |
US7062704B2 (en) * | 2001-04-30 | 2006-06-13 | Sun Microsystems, Inc. | Storage array employing scrubbing operations using multiple levels of checksums |
US7017107B2 (en) * | 2001-04-30 | 2006-03-21 | Sun Microsystems, Inc. | Storage array employing scrubbing operations at the disk-controller level |
US6839815B2 (en) * | 2001-05-07 | 2005-01-04 | Hitachi, Ltd. | System and method for storage on demand service in a global SAN environment |
US6915397B2 (en) * | 2001-06-01 | 2005-07-05 | Hewlett-Packard Development Company, L.P. | System and method for generating point in time storage copy |
US6757753B1 (en) | 2001-06-06 | 2004-06-29 | Lsi Logic Corporation | Uniform routing of storage access requests through redundant array controllers |
JP4232357B2 (en) * | 2001-06-14 | 2009-03-04 | 株式会社日立製作所 | Computer system |
JP4175788B2 (en) | 2001-07-05 | 2008-11-05 | 株式会社日立製作所 | Volume controller |
US7076510B2 (en) * | 2001-07-12 | 2006-07-11 | Brown William P | Software raid methods and apparatuses including server usage based write delegation |
US7289499B1 (en) * | 2001-07-16 | 2007-10-30 | Network Appliance, Inc. | Integrated system and method for controlling telecommunication network data communicated over a local area network and storage data communicated over a storage area network |
US7239642B1 (en) | 2001-07-16 | 2007-07-03 | Network Appliance, Inc. | Multi-protocol network interface card |
US7404206B2 (en) * | 2001-07-17 | 2008-07-22 | Yottayotta, Inc. | Network security devices and methods |
US7127565B2 (en) * | 2001-08-20 | 2006-10-24 | Spinnaker Networks, Inc. | Method and system for safely arbitrating disk drive ownership using a timestamp voting algorithm |
US7257815B2 (en) * | 2001-09-05 | 2007-08-14 | Microsoft Corporation | Methods and system of managing concurrent access to multiple resources |
US7472231B1 (en) | 2001-09-07 | 2008-12-30 | Netapp, Inc. | Storage area network data cache |
US7330892B2 (en) * | 2001-09-07 | 2008-02-12 | Network Appliance, Inc. | High-speed data transfer in a storage virtualization controller |
US20030055932A1 (en) * | 2001-09-19 | 2003-03-20 | Dell Products L.P. | System and method for configuring a storage area network |
US7864758B1 (en) | 2001-09-28 | 2011-01-04 | Emc Corporation | Virtualization in a storage system |
US6976134B1 (en) | 2001-09-28 | 2005-12-13 | Emc Corporation | Pooling and provisioning storage resources in a storage network |
US7185062B2 (en) * | 2001-09-28 | 2007-02-27 | Emc Corporation | Switch-based storage services |
US7421509B2 (en) * | 2001-09-28 | 2008-09-02 | Emc Corporation | Enforcing quality of service in a storage network |
US7558264B1 (en) | 2001-09-28 | 2009-07-07 | Emc Corporation | Packet classification in a storage system |
US7707304B1 (en) | 2001-09-28 | 2010-04-27 | Emc Corporation | Storage switch for storage area network |
US7404000B2 (en) * | 2001-09-28 | 2008-07-22 | Emc Corporation | Protocol translation in a storage system |
US20030079018A1 (en) * | 2001-09-28 | 2003-04-24 | Lolayekar Santosh C. | Load balancing in a storage network |
US7499986B2 (en) | 2001-10-04 | 2009-03-03 | International Business Machines Corporation | Storage area network methods with event notification conflict resolution |
US20030154271A1 (en) * | 2001-10-05 | 2003-08-14 | Baldwin Duane Mark | Storage area network methods and apparatus with centralized management |
US6996670B2 (en) * | 2001-10-05 | 2006-02-07 | International Business Machines Corporation | Storage area network methods and apparatus with file system extension |
US20030149762A1 (en) * | 2001-10-05 | 2003-08-07 | Knight Gregory John | Storage area network methods and apparatus with history maintenance and removal |
US20030109679A1 (en) * | 2001-10-10 | 2003-06-12 | Green Brent Everett | Flax protein isolate and production |
US6880101B2 (en) * | 2001-10-12 | 2005-04-12 | Dell Products L.P. | System and method for providing automatic data restoration after a storage device failure |
US6545872B1 (en) | 2001-10-12 | 2003-04-08 | Compaq Information Technologies Group, L.P. | Heat sink for edge connectors |
US6889345B2 (en) * | 2001-10-19 | 2005-05-03 | Hewlett-Packard Development Company, Lp. | System and method for locating a failed storage device in a data storage system |
US6920511B2 (en) * | 2001-10-19 | 2005-07-19 | Hewlett-Packard Development Company, L.P. | Method and apparatus for controlling communications in data storage complexes |
US6988136B2 (en) * | 2001-10-19 | 2006-01-17 | Hewlett-Packard Development Company, L.P. | Unified management system and method for multi-cabinet data storage complexes |
US6912599B2 (en) * | 2001-10-19 | 2005-06-28 | Hewlett-Packard Development Company, L.P. | Method and apparatus for sensing positions of device enclosures within multi-shelf cabinets |
US6895467B2 (en) | 2001-10-22 | 2005-05-17 | Hewlett-Packard Development Company, L.P. | System and method for atomizing storage |
US6931487B2 (en) * | 2001-10-22 | 2005-08-16 | Hewlett-Packard Development Company L.P. | High performance multi-controller processing |
US8046469B2 (en) * | 2001-10-22 | 2011-10-25 | Hewlett-Packard Development Company, L.P. | System and method for interfacing with virtual storage |
US6883065B1 (en) * | 2001-11-15 | 2005-04-19 | Xiotech Corporation | System and method for a redundant communication channel via storage area network back-end |
JP2003162377A (en) * | 2001-11-28 | 2003-06-06 | Hitachi Ltd | Disk array system and method for taking over logical unit among controllers |
US7194656B2 (en) * | 2001-11-28 | 2007-03-20 | Yottayotta Inc. | Systems and methods for implementing content sensitive routing over a wide area network (WAN) |
US6732201B2 (en) * | 2001-12-17 | 2004-05-04 | Lsi Logic Corporation | Hardware speed selection behind a disk array controller |
US7159080B1 (en) * | 2001-12-20 | 2007-01-02 | Network Appliance, Inc. | System and method for storing storage operating system data in switch ports |
US7650412B2 (en) | 2001-12-21 | 2010-01-19 | Netapp, Inc. | Systems and method of implementing disk ownership in networked storage |
US7296068B1 (en) | 2001-12-21 | 2007-11-13 | Network Appliance, Inc. | System and method for transfering volume ownership in net-worked storage |
US7548975B2 (en) * | 2002-01-09 | 2009-06-16 | Cisco Technology, Inc. | Methods and apparatus for implementing virtualization of storage within a storage area network through a virtual enclosure |
US7093043B2 (en) | 2001-12-27 | 2006-08-15 | Hewlett-Packard Development Company, L.P. | Data array having redundancy messaging between array controllers over the host bus |
EP1429249A3 (en) * | 2002-01-04 | 2009-09-09 | Egenera, Inc. | Virtual networking system and method in a processing system |
US7281044B2 (en) * | 2002-01-10 | 2007-10-09 | Hitachi, Ltd. | SAN infrastructure on demand service system |
US20030140128A1 (en) * | 2002-01-18 | 2003-07-24 | Dell Products L.P. | System and method for validating a network |
US6880052B2 (en) * | 2002-03-26 | 2005-04-12 | Hewlett-Packard Development Company, Lp | Storage area network, data replication and storage controller, and method for replicating data using virtualized volumes |
US6947981B2 (en) * | 2002-03-26 | 2005-09-20 | Hewlett-Packard Development Company, L.P. | Flexible data replication mechanism |
US8051197B2 (en) * | 2002-03-29 | 2011-11-01 | Brocade Communications Systems, Inc. | Network congestion management systems and methods |
US7120826B2 (en) * | 2002-03-29 | 2006-10-10 | International Business Machines Corporation | Partial mirroring during expansion thereby eliminating the need to track the progress of stripes updated during expansion |
US20040006635A1 (en) * | 2002-04-19 | 2004-01-08 | Oesterreicher Richard T. | Hybrid streaming platform |
US7899924B2 (en) * | 2002-04-19 | 2011-03-01 | Oesterreicher Richard T | Flexible streaming hardware |
US20040006636A1 (en) * | 2002-04-19 | 2004-01-08 | Oesterreicher Richard T. | Optimized digital media delivery engine |
US7328284B2 (en) * | 2002-05-06 | 2008-02-05 | Qlogic, Corporation | Dynamic configuration of network data flow using a shared I/O subsystem |
US7356608B2 (en) * | 2002-05-06 | 2008-04-08 | Qlogic, Corporation | System and method for implementing LAN within shared I/O subsystem |
US7447778B2 (en) * | 2002-05-06 | 2008-11-04 | Qlogic, Corporation | System and method for a shared I/O subsystem |
US7404012B2 (en) * | 2002-05-06 | 2008-07-22 | Qlogic, Corporation | System and method for dynamic link aggregation in a shared I/O subsystem |
JP2003330782A (en) * | 2002-05-10 | 2003-11-21 | Hitachi Ltd | Computer system |
US7010528B2 (en) * | 2002-05-23 | 2006-03-07 | International Business Machines Corporation | Mechanism for running parallel application programs on metadata controller nodes |
US7448077B2 (en) * | 2002-05-23 | 2008-11-04 | International Business Machines Corporation | File level security for a metadata controller in a storage area network |
US8140622B2 (en) | 2002-05-23 | 2012-03-20 | International Business Machines Corporation | Parallel metadata service in storage area network environment |
US6732171B2 (en) * | 2002-05-31 | 2004-05-04 | Lefthand Networks, Inc. | Distributed network storage system with virtualization |
US7024586B2 (en) * | 2002-06-24 | 2006-04-04 | Network Appliance, Inc. | Using file system information in raid data reconstruction and migration |
GB0214669D0 (en) * | 2002-06-26 | 2002-08-07 | Ibm | Method for maintaining data access during failure of a controller |
US20040024807A1 (en) * | 2002-07-31 | 2004-02-05 | Microsoft Corporation | Asynchronous updates of weakly consistent distributed state information |
US6928509B2 (en) * | 2002-08-01 | 2005-08-09 | International Business Machines Corporation | Method and apparatus for enhancing reliability and scalability of serial storage devices |
CA2494325A1 (en) * | 2002-08-02 | 2004-02-12 | Grass Valley (U.S.), Inc. | Real-time fail-over recovery for a media area network |
US7418702B2 (en) * | 2002-08-06 | 2008-08-26 | Sheng (Ted) Tai Tsao | Concurrent web based multi-task support for control management system |
US7283560B1 (en) | 2002-08-08 | 2007-10-16 | Vicom Systems, Inc. | Method and apparatus for address translation between fibre channel addresses and SCSI addresses |
US7571206B2 (en) * | 2002-08-12 | 2009-08-04 | Equallogic, Inc. | Transparent request routing for a partitioned application service |
US7134044B2 (en) * | 2002-08-16 | 2006-11-07 | International Business Machines Corporation | Method, system, and program for providing a mirror copy of data |
JP2004086721A (en) * | 2002-08-28 | 2004-03-18 | Nec Corp | Data reproducing system, relay system, data transmission/receiving method, and program for reproducing data in storage |
US8805918B1 (en) | 2002-09-11 | 2014-08-12 | Cisco Technology, Inc. | Methods and apparatus for implementing exchange management for virtualization of storage within a storage area network |
US20040085908A1 (en) * | 2002-10-31 | 2004-05-06 | Brocade Communications Systems, Inc. | Method and apparatus for managing locking of resources in a cluster by use of a network fabric |
US7627650B2 (en) * | 2003-01-20 | 2009-12-01 | Equallogic, Inc. | Short-cut response for distributed services |
US7461146B2 (en) * | 2003-01-20 | 2008-12-02 | Equallogic, Inc. | Adaptive storage block data distribution |
US7937551B2 (en) * | 2003-01-21 | 2011-05-03 | Dell Products L.P. | Storage systems having differentiated storage pools |
US20040210724A1 (en) * | 2003-01-21 | 2004-10-21 | Equallogic Inc. | Block data migration |
US8499086B2 (en) | 2003-01-21 | 2013-07-30 | Dell Products L.P. | Client load distribution |
US7127577B2 (en) * | 2003-01-21 | 2006-10-24 | Equallogic Inc. | Distributed snapshot process |
US8037264B2 (en) * | 2003-01-21 | 2011-10-11 | Dell Products, L.P. | Distributed snapshot process |
US7181574B1 (en) * | 2003-01-30 | 2007-02-20 | Veritas Operating Corporation | Server cluster using informed prefetching |
US20040181707A1 (en) * | 2003-03-11 | 2004-09-16 | Hitachi, Ltd. | Method and apparatus for seamless management for disaster recovery |
US7817583B2 (en) * | 2003-04-28 | 2010-10-19 | Hewlett-Packard Development Company, L.P. | Method for verifying a storage area network configuration |
JP2005157825A (en) * | 2003-11-27 | 2005-06-16 | Hitachi Ltd | Computer system with function to recover from failure and method for recovery from failure |
JP2005004350A (en) * | 2003-06-10 | 2005-01-06 | Sony Ericsson Mobilecommunications Japan Inc | Resource management method and device, resource management program, and storage medium |
US7065589B2 (en) | 2003-06-23 | 2006-06-20 | Hitachi, Ltd. | Three data center remote copy system with journaling |
US20050015655A1 (en) * | 2003-06-30 | 2005-01-20 | Clayton Michele M. | Intermediate station |
US7379974B2 (en) * | 2003-07-14 | 2008-05-27 | International Business Machines Corporation | Multipath data retrieval from redundant array |
US6939701B2 (en) | 2003-08-08 | 2005-09-06 | Novozymes, Inc. | Polypeptides having oxaloacetate hydrolase activity and nucleic acids encoding same |
JP4313650B2 (en) * | 2003-11-07 | 2009-08-12 | 株式会社日立製作所 | File server, redundancy recovery method, program, and recording medium |
US7934023B2 (en) * | 2003-12-01 | 2011-04-26 | Cisco Technology, Inc. | Apparatus and method for performing fast fibre channel write operations over relatively high latency networks |
US7446433B2 (en) * | 2004-01-23 | 2008-11-04 | American Power Conversion Corporation | Methods and apparatus for providing uninterruptible power |
CN100452673C (en) * | 2004-02-16 | 2009-01-14 | 上海欣国信息技术有限公司 | Digital auendant console |
JP2005275829A (en) * | 2004-03-25 | 2005-10-06 | Hitachi Ltd | Storage system |
US7529291B2 (en) * | 2004-04-13 | 2009-05-05 | Raytheon Company | Methods and structures for rapid code acquisition in spread spectrum communications |
US7409494B2 (en) | 2004-04-30 | 2008-08-05 | Network Appliance, Inc. | Extension of write anywhere file system layout |
US20050289143A1 (en) | 2004-06-23 | 2005-12-29 | Exanet Ltd. | Method for managing lock resources in a distributed storage system |
WO2006014573A2 (en) * | 2004-07-07 | 2006-02-09 | Yotta Yotta, Inc. | Systems and methods for providing distributed cache coherence |
CN101027668B (en) * | 2004-07-21 | 2012-01-04 | 海滩无极限有限公司 | Distributed storage architecture based on block map caching and VFS stackable file system modules |
US7412545B2 (en) * | 2004-07-22 | 2008-08-12 | International Business Machines Corporation | Apparatus and method for updating I/O capability of a logically-partitioned computer system |
KR100868820B1 (en) * | 2004-07-23 | 2008-11-14 | 비치 언리미티드 엘엘씨 | A method and system for communicating a data stream and a method of controlling a data storage level |
US7240155B2 (en) * | 2004-09-30 | 2007-07-03 | International Business Machines Corporation | Decision mechanisms for adapting RAID operation placement |
US7529967B2 (en) * | 2004-11-04 | 2009-05-05 | Rackable Systems Inc. | Method and system for network storage device failure protection and recovery |
US7535832B2 (en) * | 2004-11-22 | 2009-05-19 | International Business Machines Corporation | Apparatus and method to set the signaling rate of a switch domain disposed within an information storage and retrieval system |
US9495263B2 (en) * | 2004-12-21 | 2016-11-15 | Infortrend Technology, Inc. | Redundant SAS storage virtualization subsystem and system using the same, and method therefor |
JP4563794B2 (en) * | 2004-12-28 | 2010-10-13 | 株式会社日立製作所 | Storage system and storage management method |
CN100373354C (en) * | 2005-01-20 | 2008-03-05 | 英业达股份有限公司 | Data accessing system and method capable of recognizing disk cache content validity |
US7401260B2 (en) * | 2005-01-28 | 2008-07-15 | International Business Machines Corporation | Apparatus, system, and method for performing storage device maintenance |
US7535917B1 (en) | 2005-02-22 | 2009-05-19 | Netapp, Inc. | Multi-protocol network adapter |
JP4815825B2 (en) * | 2005-03-10 | 2011-11-16 | 日本電気株式会社 | Disk array device and method for reconstructing the same |
US7676688B1 (en) * | 2005-03-16 | 2010-03-09 | Symantec Corporation | Concurrent data broadcast of selected common groups of data blocks |
US7600214B2 (en) * | 2005-04-18 | 2009-10-06 | Broadcom Corporation | Use of metadata for seamless updates |
CA2615324A1 (en) * | 2005-07-14 | 2007-07-05 | Yotta Yotta, Inc. | Maintaining write order fidelity on a multi-writer system |
US8819088B2 (en) * | 2005-07-14 | 2014-08-26 | International Business Machines Corporation | Implementing storage management functions using a data store system |
CN100405313C (en) * | 2005-07-22 | 2008-07-23 | 鸿富锦精密工业(深圳)有限公司 | Link control card detecting system and method |
US7502957B2 (en) * | 2005-09-09 | 2009-03-10 | International Business Machines Corporation | Method and system to execute recovery in non-homogeneous multi processor environments |
US8396966B2 (en) * | 2005-10-26 | 2013-03-12 | Hewlett-Packard Development Company, L.P. | Method and an apparatus for creating visual representations of farms that enables connecting farms securely |
US9118698B1 (en) | 2005-12-02 | 2015-08-25 | Branislav Radovanovic | Scalable data storage architecture and methods of eliminating I/O traffic bottlenecks |
US8347010B1 (en) * | 2005-12-02 | 2013-01-01 | Branislav Radovanovic | Scalable data storage architecture and methods of eliminating I/O traffic bottlenecks |
US8275949B2 (en) * | 2005-12-13 | 2012-09-25 | International Business Machines Corporation | System support storage and computer system |
CA2642145C (en) | 2006-02-14 | 2013-09-24 | Yottayotta, Inc. | Systems and methods for obtaining ultra-high data availability and geographic disaster tolerance |
US7676702B2 (en) * | 2006-08-14 | 2010-03-09 | International Business Machines Corporation | Preemptive data protection for copy services in storage systems and applications |
JP5179031B2 (en) * | 2006-09-13 | 2013-04-10 | 株式会社日立製作所 | Storage system that makes effective use of available ports |
US20080159277A1 (en) * | 2006-12-15 | 2008-07-03 | Brocade Communications Systems, Inc. | Ethernet over fibre channel |
US20080181243A1 (en) * | 2006-12-15 | 2008-07-31 | Brocade Communications Systems, Inc. | Ethernet forwarding in high performance fabrics |
US7882393B2 (en) * | 2007-03-28 | 2011-02-01 | International Business Machines Corporation | In-band problem log data collection between a host system and a storage system |
US7779308B2 (en) * | 2007-06-21 | 2010-08-17 | International Business Machines Corporation | Error processing across multiple initiator network |
US8892942B2 (en) * | 2007-07-27 | 2014-11-18 | Hewlett-Packard Development Company, L.P. | Rank sparing system and method |
US20090060503A1 (en) * | 2007-09-05 | 2009-03-05 | Nec Laboratories America, Inc. | Storage over optical/wireless integrated broadband access network (soba) architecture |
US8583780B2 (en) * | 2007-11-20 | 2013-11-12 | Brocade Communications Systems, Inc. | Discovery of duplicate address in a network by reviewing discovery frames received at a port |
US8108454B2 (en) * | 2007-12-17 | 2012-01-31 | Brocade Communications Systems, Inc. | Address assignment in Fibre Channel over Ethernet environments |
JP5072692B2 (en) * | 2008-04-07 | 2012-11-14 | 株式会社日立製作所 | Storage system with multiple storage system modules |
US8347182B2 (en) * | 2008-07-01 | 2013-01-01 | International Business Machines Corporation | Ensuring data consistency |
US20100043006A1 (en) * | 2008-08-13 | 2010-02-18 | Egenera, Inc. | Systems and methods for a configurable deployment platform with virtualization of processing resource specific persistent settings |
US20100083268A1 (en) * | 2008-09-29 | 2010-04-01 | Morris Robert P | Method And System For Managing Access To A Resource By A Process Processing A Media Stream |
US8086911B1 (en) * | 2008-10-29 | 2011-12-27 | Netapp, Inc. | Method and apparatus for distributed reconstruct in a raid system |
US7882389B2 (en) * | 2008-11-18 | 2011-02-01 | International Business Machines Corporation | Dynamic reassignment of devices attached to redundant controllers |
US8495417B2 (en) * | 2009-01-09 | 2013-07-23 | Netapp, Inc. | System and method for redundancy-protected aggregates |
US8848575B2 (en) | 2009-02-23 | 2014-09-30 | Brocade Communications Systems, Inc. | High availability and multipathing for fibre channel over ethernet |
US8719829B2 (en) * | 2009-03-09 | 2014-05-06 | International Business Machines Corporation | Synchronizing processes in a computing resource by locking a resource for a process at a predicted time slot |
US20100293145A1 (en) * | 2009-05-15 | 2010-11-18 | Hewlett-Packard Development Company, L.P. | Method of Selective Replication in a Storage Area Network |
US8074003B1 (en) * | 2009-12-28 | 2011-12-06 | Emc Corporation | Host-based storage controller providing block devices in geographically distributed storage |
CN102209097A (en) * | 2010-03-31 | 2011-10-05 | 英业达股份有限公司 | System for allocating storage resources of storage local area network |
US8738724B2 (en) | 2010-05-25 | 2014-05-27 | Microsoft Corporation | Totally ordered log on appendable storage |
WO2012020505A1 (en) * | 2010-08-13 | 2012-02-16 | 富士通株式会社 | Memory control device, information processing device, and control method for a memory control device |
CN102129400B (en) | 2010-12-29 | 2013-12-04 | 华为数字技术(成都)有限公司 | Storage system connection configuration method and equipment and storage system |
CN104094603B (en) * | 2011-12-28 | 2018-06-08 | 英特尔公司 | For carrying out the system and method that integrated metadata is inserted into video coding system |
CN103838515B (en) * | 2012-11-23 | 2016-08-03 | 中国科学院声学研究所 | A kind of method and system of server cluster access scheduling multi-controller disk array |
WO2014101108A1 (en) * | 2012-12-28 | 2014-07-03 | 华为技术有限公司 | Caching method for distributed storage system, node and computer readable medium |
CN103701925B (en) * | 2013-12-31 | 2017-04-05 | 北京网康科技有限公司 | Source synchronous management-control method |
US9547448B2 (en) * | 2014-02-24 | 2017-01-17 | Netapp, Inc. | System and method for transposed storage in raid arrays |
US10455019B2 (en) * | 2014-09-10 | 2019-10-22 | Oracle International Corporation | Highly performant reliable message storage using in-memory replication technology |
US9853873B2 (en) | 2015-01-10 | 2017-12-26 | Cisco Technology, Inc. | Diagnosis and throughput measurement of fibre channel ports in a storage area network environment |
US9900250B2 (en) | 2015-03-26 | 2018-02-20 | Cisco Technology, Inc. | Scalable handling of BGP route information in VXLAN with EVPN control plane |
US10222986B2 (en) | 2015-05-15 | 2019-03-05 | Cisco Technology, Inc. | Tenant-level sharding of disks with tenant-specific storage modules to enable policies per tenant in a distributed storage system |
US11588783B2 (en) | 2015-06-10 | 2023-02-21 | Cisco Technology, Inc. | Techniques for implementing IPV6-based distributed storage space |
US10778765B2 (en) | 2015-07-15 | 2020-09-15 | Cisco Technology, Inc. | Bid/ask protocol in scale-out NVMe storage |
US9892075B2 (en) | 2015-12-10 | 2018-02-13 | Cisco Technology, Inc. | Policy driven storage in a microserver computing environment |
CN105677499B (en) * | 2015-12-29 | 2018-10-12 | 曙光信息产业(北京)有限公司 | Hardware based time-out management platform |
CN105786656B (en) * | 2016-02-17 | 2019-08-13 | 中科院成都信息技术股份有限公司 | Redundant array of independent disks disaster tolerance storage method based on random matrix |
US10331353B2 (en) | 2016-04-08 | 2019-06-25 | Branislav Radovanovic | Scalable data access system and methods of eliminating controller bottlenecks |
US10140172B2 (en) | 2016-05-18 | 2018-11-27 | Cisco Technology, Inc. | Network-aware storage repairs |
US20170351639A1 (en) | 2016-06-06 | 2017-12-07 | Cisco Technology, Inc. | Remote memory access using memory mapped addressing among multiple compute nodes |
US10664169B2 (en) | 2016-06-24 | 2020-05-26 | Cisco Technology, Inc. | Performance of object storage system by reconfiguring storage devices based on latency that includes identifying a number of fragments that has a particular storage device as its primary storage device and another number of fragments that has said particular storage device as its replica storage device |
US11563695B2 (en) | 2016-08-29 | 2023-01-24 | Cisco Technology, Inc. | Queue protection using a shared global memory reserve |
CN106527978B (en) * | 2016-10-19 | 2019-07-09 | 华中科技大学 | A kind of multi-controller implementation method based on cyclic annular virtual dual control |
US10545914B2 (en) | 2017-01-17 | 2020-01-28 | Cisco Technology, Inc. | Distributed object storage |
US10243823B1 (en) | 2017-02-24 | 2019-03-26 | Cisco Technology, Inc. | Techniques for using frame deep loopback capabilities for extended link diagnostics in fibre channel storage area networks |
US10713203B2 (en) | 2017-02-28 | 2020-07-14 | Cisco Technology, Inc. | Dynamic partition of PCIe disk arrays based on software configuration / policy distribution |
US10254991B2 (en) | 2017-03-06 | 2019-04-09 | Cisco Technology, Inc. | Storage area network based extended I/O metrics computation for deep insight into application performance |
CN109089255B (en) * | 2017-06-14 | 2022-01-25 | 中国移动通信有限公司研究院 | User position notification control method, device, system, equipment and storage medium |
US10303534B2 (en) | 2017-07-20 | 2019-05-28 | Cisco Technology, Inc. | System and method for self-healing of application centric infrastructure fabric memory |
US10404596B2 (en) | 2017-10-03 | 2019-09-03 | Cisco Technology, Inc. | Dynamic route profile storage in a hardware trie routing table |
US10942666B2 (en) | 2017-10-13 | 2021-03-09 | Cisco Technology, Inc. | Using network device replication in distributed storage clusters |
Family Cites Families (27)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5671436A (en) | 1991-08-21 | 1997-09-23 | Norand Corporation | Versatile RF data capture system |
WO1993018456A1 (en) * | 1992-03-13 | 1993-09-16 | Emc Corporation | Multiple controller sharing in a redundant storage array |
US5459857A (en) * | 1992-05-15 | 1995-10-17 | Storage Technology Corporation | Fault tolerant disk array data storage subsystem |
US5611049A (en) | 1992-06-03 | 1997-03-11 | Pitts; William M. | System for accessing distributed data cache channel at each network node to pass requests and data |
US5758058A (en) * | 1993-03-31 | 1998-05-26 | Intel Corporation | Apparatus and method for initializing a master/checker fault detecting microprocessor |
US5574851A (en) * | 1993-04-19 | 1996-11-12 | At&T Global Information Solutions Company | Method for performing on-line reconfiguration of a disk array concurrent with execution of disk I/O operations |
US5694581A (en) * | 1993-09-07 | 1997-12-02 | Industrial Technology Research Institute | Concurrent disk array management system implemented with CPU executable extension |
TW252248B (en) | 1994-08-23 | 1995-07-21 | Ibm | A semiconductor memory based server for providing multimedia information on demand over wide area networks |
US6085234A (en) | 1994-11-28 | 2000-07-04 | Inca Technology, Inc. | Remote file services network-infrastructure cache |
US5657468A (en) | 1995-08-17 | 1997-08-12 | Ambex Technologies, Inc. | Method and apparatus for improving performance in a reduntant array of independent disks |
US5875456A (en) | 1995-08-17 | 1999-02-23 | Nstor Corporation | Storage device array and methods for striping and unstriping data and for adding and removing disks online to/from a raid storage array |
US5826001A (en) * | 1995-10-13 | 1998-10-20 | Digital Equipment Corporation | Reconstructing data blocks in a raid array data storage system having storage device metadata and raid set metadata |
US5862312A (en) * | 1995-10-24 | 1999-01-19 | Seachange Technology, Inc. | Loosely coupled mass storage computer cluster |
JP2848300B2 (en) * | 1995-12-27 | 1999-01-20 | 日本電気株式会社 | Nonvolatile semiconductor memory device |
US6073218A (en) * | 1996-12-23 | 2000-06-06 | Lsi Logic Corp. | Methods and apparatus for coordinating shared multiple raid controller access to common storage devices |
JPH10289524A (en) * | 1997-04-11 | 1998-10-27 | Sony Corp | Recording medium drive device |
US6151297A (en) | 1997-07-08 | 2000-11-21 | Hewlett-Packard Company | Method and system for link level server/switch trunking |
US6049833A (en) | 1997-08-29 | 2000-04-11 | Cisco Technology, Inc. | Mapping SNA session flow control to TCP flow control |
US6216173B1 (en) | 1998-02-03 | 2001-04-10 | Redbox Technologies Limited | Method and apparatus for content processing and routing |
US6138247A (en) * | 1998-05-14 | 2000-10-24 | Motorola, Inc. | Method for switching between multiple system processors |
US6243829B1 (en) * | 1998-05-27 | 2001-06-05 | Hewlett-Packard Company | Memory controller supporting redundant synchronous memories |
US6148414A (en) * | 1998-09-24 | 2000-11-14 | Seek Systems, Inc. | Methods and systems for implementing shared disk array management functions |
US6405219B2 (en) | 1999-06-22 | 2002-06-11 | F5 Networks, Inc. | Method and system for automatically updating the version of a set of files stored on content servers |
US7343413B2 (en) | 2000-03-21 | 2008-03-11 | F5 Networks, Inc. | Method and system for optimizing a network by independently scaling control segments and data flow |
US7299290B2 (en) | 2000-03-22 | 2007-11-20 | Yottayotta, Inc. | Method and system for providing multimedia information on demand over wide area networks |
US8281022B1 (en) | 2000-06-30 | 2012-10-02 | Emc Corporation | Method and apparatus for implementing high-performance, scaleable data processing and storage systems |
US6658478B1 (en) | 2000-08-04 | 2003-12-02 | 3Pardata, Inc. | Data storage system |
-
1999
- 1999-03-03 US US09/261,906 patent/US6148414A/en not_active Expired - Lifetime
-
2000
- 2000-02-08 EP EP07104609A patent/EP1796001B1/en not_active Expired - Lifetime
- 2000-02-08 CN CN00804532.1A patent/CN100489796C/en not_active Expired - Lifetime
- 2000-02-08 EP EP00911736A patent/EP1171820B1/en not_active Expired - Lifetime
- 2000-02-08 DE DE60042225T patent/DE60042225D1/en not_active Expired - Lifetime
- 2000-02-08 AT AT00911736T patent/ATE359552T1/en not_active IP Right Cessation
- 2000-02-08 CA CA2363726A patent/CA2363726C/en not_active Expired - Lifetime
- 2000-02-08 WO PCT/US2000/003275 patent/WO2000052576A1/en active IP Right Grant
- 2000-02-08 NZ NZ513789A patent/NZ513789A/en unknown
- 2000-02-08 KR KR1020017011191A patent/KR20020012539A/en not_active Application Discontinuation
- 2000-02-08 AT AT07104609T patent/ATE431589T1/en not_active IP Right Cessation
- 2000-02-08 DE DE60034327T patent/DE60034327T2/en not_active Expired - Lifetime
- 2000-02-08 JP JP2000602929A patent/JP2002538549A/en active Pending
- 2000-02-08 AU AU33586/00A patent/AU3358600A/en not_active Abandoned
- 2000-03-20 TW TW089103707A patent/TW468107B/en not_active IP Right Cessation
- 2000-09-07 US US09/657,258 patent/US6912668B1/en not_active Expired - Lifetime
-
2005
- 2005-03-15 US US11/082,178 patent/US7246260B2/en not_active Expired - Lifetime
Also Published As
Publication number | Publication date |
---|---|
CN100489796C (en) | 2009-05-20 |
CA2363726C (en) | 2010-06-29 |
KR20020012539A (en) | 2002-02-16 |
EP1171820A1 (en) | 2002-01-16 |
EP1796001A2 (en) | 2007-06-13 |
WO2000052576A9 (en) | 2001-08-30 |
ATE359552T1 (en) | 2007-05-15 |
CN1350674A (en) | 2002-05-22 |
AU3358600A (en) | 2000-09-21 |
EP1796001A3 (en) | 2007-06-27 |
DE60034327T2 (en) | 2008-01-03 |
US6912668B1 (en) | 2005-06-28 |
US6148414A (en) | 2000-11-14 |
TW468107B (en) | 2001-12-11 |
JP2002538549A (en) | 2002-11-12 |
US7246260B2 (en) | 2007-07-17 |
DE60042225D1 (en) | 2009-06-25 |
ATE431589T1 (en) | 2009-05-15 |
US20060005076A1 (en) | 2006-01-05 |
NZ513789A (en) | 2003-10-31 |
WO2000052576A1 (en) | 2000-09-08 |
EP1171820B1 (en) | 2007-04-11 |
EP1171820A4 (en) | 2005-06-15 |
DE60034327D1 (en) | 2007-05-24 |
EP1796001B1 (en) | 2009-05-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
CA2363726A1 (en) | Methods and systems for implementing shared disk array management functions | |
US6928514B2 (en) | Method and apparatus for teaming storage controllers | |
US7266706B2 (en) | Methods and systems for implementing shared disk array management functions | |
JP2008004120A (en) | Direct access storage system | |
US6633962B1 (en) | Method, system, program, and data structures for restricting host access to a storage space | |
CN110825670A (en) | Managed exchange between a host and a Solid State Drive (SSD) based on NVMe protocol | |
US7340551B2 (en) | Bridge permitting access by multiple hosts to a single ported storage drive | |
CN111712802A (en) | Virtual storage drive management in a data storage system | |
US10872036B1 (en) | Methods for facilitating efficient storage operations using host-managed solid-state disks and devices thereof | |
CN100535867C (en) | Apparatus and method for file-level replication between two or more non-symmetric storage sites | |
US6851023B2 (en) | Method and system for configuring RAID subsystems with block I/O commands and block I/O path | |
US10853203B2 (en) | Storage aggregate restoration | |
US20040107176A1 (en) | Method and apparatus for providing storage control in a network of storage controllers | |
US7434022B1 (en) | Distributed workflow techniques | |
US7743180B2 (en) | Method, system, and program for managing path groups to an input/output (I/O) device | |
US10216652B1 (en) | Split target data transfer | |
US20060036790A1 (en) | Method, system, and program for returning attention to a processing system requesting a lock | |
US8392568B2 (en) | Computer system and method of managing single name space | |
US20200341692A1 (en) | Concurrent tape access | |
EP1172728A2 (en) | Information management engine and method | |
US20030188096A1 (en) | Distributing access rights to mass storage | |
JP3544219B2 (en) | File sharing system | |
JP2007128275A (en) | Automatic endian conversion reading method, writing method and copying method for data in storage |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
EEER | Examination request | ||
MKEX | Expiry |
Effective date: 20200210 |